ENQUIRE PROJECT DETAILS BY GENERAL PUBLIC

Project Details
Funding Scheme : Early Career Scheme
Project Number : 27200520
Project Title(English) : From point cloud to building and city information model (BIM/CIM): A study of architectonic grammar optimization 
Project Title(Chinese) : 建築及城市信息模型的一類構造語法最優化方法 
Principal Investigator(English) : Prof Xue, Fan 
Principal Investigator(Chinese) :  
Department : Department of Real Estate and Construction
Institution : The University of Hong Kong
E-mail Address : xuef@hku.hk 
Tel : 3917 4174 
Co - Investigator(s) :
Panel : Engineering
Subject Area : Civil Engineering, Surveying, Building & Construction
Exercise Year : 2020 / 21
Fund Approved : 635,569
Project Status : Completed
Completion Date : 31-12-2023
Project Objectives :
To define the architectonic grammar in building and city scenes;
To formulate building/city information model (BIM/CIM) reconstruction from 3D point clouds into a nonlinear architectonic grammar optimization (AGO) problem;
To apply and revise state-of-the-art derivative-free optimization (DFO) algorithms for solving AGO problems; and
To integrate the proposed AGO approach to form a generalizable methodology and disseminate it with educational and professional software.
Abstract as per original application
(English/Chinese):
本項目旨在研究一類系統化化的方法，將三維點雲數據重建為建築及城市信息模型（BIM/CIM）。構造語法（architectonic grammar）不僅普遍存在于建築設計語言中，也是建築及城市信息模型的數據語言的基本標準。本研究將以構造語法最優化的視角來探索模型重建的問題。首先，定義構造語法來反應建築及城市信息模型，主要包括多種構件模型在參數化建模配置中的語義（例如類型、尺寸及顔色）及其文法規則（例如拓撲關係及多樣性）。然後，將構造語法設計映射至點雲數據匹配，建立一類非綫性優化數學模型，從而剋服點雲中的噪音、遮蔽及雜物等數據缺陷。第三步，研究運用高階無導數優化算法計算該非綫性優化問題，求解出最符合輸入數據和構造語法集的參數化構造方案。最終，最優化的參數化構造方案將剋服點雲數據缺陷，並自動重建出準確的建築及城市信息模型。研究成果將為處理非結構化點雲數據的相關研究和實踐提供一種新的自動化手段，並能為建築和建設的課堂教學提供前沿討論案例。
Realisation of objectives: 5.1 Scope of investigation -- Building/city information modeling (BIM/CIM) is vital to smart and sustainable development in the digital twin era. However, the BIM/CIM of existing buildings is hindered by the imperfections of 3D measurement data, such as noises, occlusions, and clutter in point clouds. The research team notices that buildings/cities' innate architectonic designs may overcome some data imperfections. -- This project studies architectonic grammar optimization methods to find the best design for BIM/CIM reconstruction. -- We define a nonlinear problem of architectonic grammar by integrating the global features (e.g., symmetry, enclosure, and urban patterns) and local features (e.g., local section backbones, deep learning-based semantics) in architectonic grammar and conventional 3D geometric error metrics. -- We apply heuristic optimization, DFO (derivative-free optimization), and deep learning algorithms to realize an automatic “best-fit” BIM/CIM reconstruction and apply the grammar/skeleton-guided model back to BIM/CIM to analyze and improve the overall performance. 5.2 Objectives achieved -- Objective 1 was achieved [100%]. The focused architectonic grammar includes global features and local features, sometimes the shape grammars can be inherited from established data sources according to [4]. We have identified and presented the foundation, example grammar, three global features, and two local features in the following publications: o Paper [9] [Note: #9 in Sect. 8]: A novel Building Section Skeleton theory as a foundation of architectonic grammar. o Paper [10]: A Symmetric Skeleton Grammar was defined and formulated. o Optional architectonic features for the grammar: -->> Paper [5]: Architectural symmetry for BIM/CIM, which won a Merit Award of Research in The Hong Kong OpenBIM / OpenGIS Awards 2022, -->> [Award #3 in Sect. 11]: enclosure of indoor space for BIM, -->> [3]: acoustic patterns of indoor settings for BIM, -->> Papers [2,8,11]: deep transfer learning-based semantics -->> Objective 2 was achieved sustainably [95%]. In addition to the general architectonic grammar optimization formulation in the proposal, Chapter [1] also defines a level of automation for architectonic optimization in the case of historical BIM. Papers [7, 10] extended the general formulation to building floor plan and individual building elements. Conference papers [3, 4, 2, 7] showcased the extended formulations for further BIM/CIM scenarios. -- Objective 3 was achieved sustainably [90%]. Papers [3,5,10] have demonstrated that DFO algorithms can successfully solve the architectonic optimization and its global/local features. Papers [2,8,11] also confirmed deep learning can be transferred for semantic segmentation for BIM/CIM, in terms of both 2D images and 3D point clouds for BIM/CIM reconstruction. -- Objective 4 was achieved [100%]. The approach was integrated and applied to several related problems. First, Paper [9] presented a state-of-the-art (SOTA) BIM/CIM compact reconstruction that bested all existing methods in terms of error and model compactness. The integrated approach has been piloted in a taught MSc course RECO7613 in 2021-2024, and in BSc course RECO2041 in 2022-2024. Grammar-guided parametric 3D reconstruction was taught in a popular gamified education environment, Minecraft Education Edition (Conference paper [6]). Our BIM reconstruction methods on architectonic regularities and enclosure won 6 awards from 2nd to 4th International Scan-to-BIM Challenge, CVPR 2022-2024[Awards 3, 10, 13 in Sect. 11]. Some papers have opened their source codes to the public via HKU Library and GitHub. 5.3 Deviations from the original plan -- There was a minor deviation in achieving Objective 3. We realized not only DFO algorithms but also heuristics optimization and deep learning algorithms for solving the architectonic grammar optimization formulations. The reason was that the optimization would be too sophisticated to be solved solely by DFO, if the scale of 3D point clouds went large. 5.4 Extensions and success beyond the original plan Apart from the successful execution of the project, we also looked into a few topics related to the project. -- Journal papers [2,11] proposed a 3D window view assessment based on CIM. -- Chapters [6, 1] extended the architectonic concept to digital heritage. -- Journal papers [10,8] applied the symmetric skeleton grammar and associated deep features to BIM design and details reconstruction.
Summary of objectives addressed:
Objectives Addressed Percentage achieved
1.To define the architectonic grammar in building and city scenesYes100%
2.To formulate building/city information model (BIM/CIM) reconstruction from 3D point clouds into a nonlinear architectonic grammar optimization (AGO) problemYes95%
3.To apply and revise state-of-the-art derivative-free optimization (DFO) algorithms for solving AGO problemsYes90%
4.To integrate the proposed AGO approach to form a generalizable methodology and disseminate it with educational and professional softwareYes100%
N/A
Research Outcome
Major findings and research outcome: 6.1.1 Contributions to the body of knowledge -- A new Building Section Skeleton (BSS) theory [9]. -- Two literature reviews on reconstruction and semantic enrichment to BIM/CIM [3,4], which attracted >100 citations -- New BIM/CIM reconstruction methods (RegARD, SSG--MMO) [5, 10] and deep learning dataset (HRHD--HK) [conf. 5], which attracted >70 citations -- Algorithmic experiments and sensitivity analysis for various (modern, rural, heritage, artefacts, and infrastructure in Hong Kong) real-world buildings and urban areas. -- Extended skeletons and grammars were applied to improve compact BIM/CIM reconstruction, city’s 3D window view, building energy performance, digital heritage, etc. 6.1.2 Research outcomes: With citations, impact factors, and honors 8 journal papers, which attracted >250 citations -- [2] Citation = 40 (as of Jan 2025) Journal impact factor = 7.9 (JCR 2024) -- [4] Citation = 104 Journal impact factor = 8.0 -- [5] Citation = 54 Journal impact factor = 4.2 -- [7] Citation = 2 Journal impact factor = 9.6 -- [8] Citation = 24 Journal impact factor = 9.6 -- [9] Citation = 5 Journal impact factor = 10.6 -- [10] Citation = 16 Journal impact factor = 6.6 -- [11] Citation = 14 Journal impact factor = 7.9 16 International/local research awards received: -- Please refer to Section 12. Note that each entry of #3, 10, and 13 contains two awards. 8 International conference papers; 6 invited lectures for HKPolyU, Tsinghua, Huazhong Univ of Science and Tech, South China Univ of Tech, Tongji, and Leica Geosciences between 2021 and 2024.
Potential for further development of the research
and the proposed course of action:		 The findings of the project have successfully enabled a TRS project, a CRF project, one ITF project, a Shenzhen-Hong Kong-Macau TRP project, and a Guangdong Natural Science Funding project: -- TRS (T22-504/21-R, HK$34M) on generative urban design facilitated by optimization. -- CRF (C7080-22GF) on generative building design facilitated by optimization -- ITF ITSP platform project (2019B010151001) (PC, HK$ 5.2 million) -- Shenzhen-Hong Kong-Macau TRP project (PI, HK$ 1,26million), -- Guangdong Natural Science Funding (PI, HK$ 0.1 million) In addition, the concept, methodology, cases, and findings have been taught in a set of BSc and MSc courses (RECO2041, RECO3032, RECO4012, RECO7097, RECO7613) at the University of Hong Kong.
Layman's Summary of
Completion Report:		 The research project aimed to enhance building/city information modeling (BIM/CIM) reconstruction from 3D point clouds for smart and sustainable development. We focused on resolving the imperfections in existing buildings' 3D point cloud data using architectnoic grammar. The team studied a fundamental theory of 'building section skeleton.' We then investigated architectonic grammar optimization and studied global features like urban patterns and local features like deep learning-based semantics. We combined derivative-free optimization, heuristic optimization, and deep learning algorithms to achieve an automatic "best-fit" BIM/CIM reconstruction. Key objectives were met. The outcomes were validated through experiments in different fields, peer-reviewed as academic publications (8 journal papers), integrated into educational courses, and attracted 16 peer-reviewed awards. The project also extended its impact from BIM/CIM to digital heritage and 3D window view assessment. In summary, the funded project outcomes can contribute to the body of knowledge with new theories, methods, and datasets.
Research Output
Peer-reviewed journal publication(s)
arising directly from this research project :
(* denotes the corresponding author)
Year of
Publication		 Author(s) Title and Journal/Book Accessible from Institution Repository
2021 Jing Zhang, Maosu Li, Wenjin Zhang, Yijie Wu, and Fan Xue*  Prospect of Architectonic Grammar Reconstruction from Dense 3D Point Clouds: Historical Building Information Modeling (HBIM) of Guangdong Cultural Heritages [Merit Paper Award]  No 
2022 Li, M., Xue, F.*, Wu, Y., Yeh, A.G.O.  A room with a view: Automatic assessment of window views for high-rise high-density areas using City Information Models and deep transfer learning  No 
2022 Fan Xue*  As-built building information modeling: joint effort of 3D reconstruction and semantic enrichment  No 
2021 Xue, F., Wu, L.*, Lu, W.  Semantic enrichment of building and city information models: A ten year review  No 
2021 Yijie Wu, Jianga Shang, Fan Xue*  RegARD: Symmetry-Based Coarse Registration of Smartphone’s Colorful Point Clouds with CAD Drawings for Low-Cost Digital Twin Buildings  No 
2024 Meng, S., Xu, G., Zhang, W., Xue, F.*  Decoding the past: A Genetic Algorithm-based method for extract decorative patterns in Digital Twin Heritages  No 
2024 Liang, D., Xue, F.*  4D point cloud-based spatial-temporal semantic registration for monitoring mobile crane construction activities  No 
2023 Chen, S.-H.; Xue, F.*  Automatic BIM detailing using deep features of 3D views  No 
2024 Wu, Y., Xue, F.*, Li, M., & Chen, S.-H.  A novel Building Section Skeleton for compact 3D reconstruction from point clouds: A study of high-density urban scenes  No 
2023 Qianyun Zhou, Fan Xue*  Pushing the boundaries of modular-integrated construction: A symmetric skeleton grammar-based multi-objective optimization of passive design for energy savings and daylight autonomy  No 
2023 Li, M., Xue, F.*, & Yeh, A.G.O.  Bi-objective analytics of 3D visual-physical nature exposures in high-rise high-density cities for landscape and urban planning  No 
Recognized international conference(s)
in which paper(s) related to this research
project was/were delivered :
Month/Year/City Title Conference Name
Chongqing, China A Review of As-Built BIM Using LiDAR Point Clouds  The 7th National BIM Academic Conference of China 
Dubai, UAE CIM-enabled quantitative view assessment in architectural design and space planning  the 38th International Symposium on Automation and Robotics in Construction (ISARC2021), [Plenary talk] 
Heraklion, Crete, Greece Towards fully automatic Scan-to-BIM: A prototype method integrating deep neural networks and architectonic grammar  2023 European Conference on Computing in Construction cum 40th International CIB W78 Conference 
Hong Kong Towards detailed building typology by urban-scale 3D building decomposition  The 1st International Workshop on Remote Sensing Intelligent Mapping (RSIM) 
Kuala Lumpur, Malaysia HRHD-HK: A benchmark dataset of high-rise and high-density urban scenes for 3D semantic segmentation of photogrammetric point clouds  2023 IEEE International Conference on Image Processing Challenges and Workshops 
Heraklion, Crete, Greece Invigorating AEC education using Minecraft: A case of LiDAR surveying and virtual learning  2023 European Conference on Computing in Construction cum 40th International CIB W78 Conference 
Xi'an, China BSS-Indoor: Volumetric wall reconstruction using Building Section Skeleton  29th International Symposium on Advancement of Construction Management and Real Estate 
Hong Kong Bridging the gap between point clouds for GeoAI: Role of supervised, reinforced and unsupervised learning  Workshop on GeoAI and Big Data for Urban, Environment, and Sustainability cum Inauguration Ceremony for Research Centre for Artificial Intelligence in Geomatics 
Other impact
(e.g. award of patents or prizes,
collaboration with other research institutions,
technology transfer, etc.):
Realisation of the education plan:
  SCREEN ID: SCRRM00542