The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLII-2/W17, 20196th International Workshop LowCost 3D – Sensors, Algorithms, Applications, 2–3 December 2019, Strasbourg, FranceTHE COMBINATION OF TERRESTRIAL LIDAR AND UAV PHOTOGRAMMETRYFOR INTERACTIVE ARCHITECTURAL HERITAGE VISUALIZATION USING UNITY3D GAME ENGINER. Andaru 1,2, B. K. Cahyono 1, G. Riyadi 3, Istarno 1, Djurdjani 1 , G. R. Ramadhan 4, S.Tuntas 41 Department of Geodetic Engineering, Universitas Gadjah Mada, Indonesia – (ruliandaru, bambangkun, istarno,djurdjani) Department of Geomatics, National Cheng Kung University, Taiwan3 Department of Earth Technology, Vocational School of Universitas Gadjah Mada, Indonesia - [email protected] Alumni Department of Geodetic Engineering, Universitas Gadjah Mada, Indonesia – gilangrr11, [email protected] IIKEYWORDS: Terrestrial Lidar, UAV Photogrammetry, architectural heritage, unity 3D, game engineABSTRACT:The digital 3D documentation of architectural heritage using advanced 3D measurement technologies such as UAV photogrammetryand terrestrial LiDAR (TLS) becomes a potential and efficient method since it can produce 3D pointclouds in detail and high densityof pointclouds levels. However, TLS is unable to scan the roof part of tall building, whereas UAV photogrammetry achieves highdensity of pointclouds at that area. In order to make a complete 3D pointclouds of heritage building, we merged and integrated theTLS and UAV pointclouds data by using Iterative Closest Point (ICP) algorithms into one reference system. In this study, we collectedtwo architectural heritage building in Yogyakarta, Indonesia, i.e., "Vredeburg Fort Museum (VFM)" and "Kotagede Great Mosque(KGM)", the oldest mosque in Yogyakarta. For the data acquisition, we used Faro Focus X330 and GLS 2000 Laser Scanner. Weproduced three-dimensional point clouds from UAV imagery by using Structure from Motion and Multi View Stereo (SfM-MVS)technique through Photoscan software. In order to merging and integrating both of pointclouds data, Maptek I-Site Studio 6.1 withEducational License was used. Those data were successfully registered, and according to the registration report, we had observed 20.60mm of RMS error. The 3D models and their textures in outdoor and indoor side were processed using Autodesk software. Modellingwas carried out on the structure of building’s façade base on simple geometric primitive as planes, straight lines, circles, spheres andcylinder. For interactive visualization, a modern and widely accessible game engine technology (Unity3D) was used. The result wasan interactive displaying 3D model of an architectural heritage building in LOD3 level with spatial function for measuring the size anddimension, as well as the area of object. Finally, we created the online version of interactive 3D viewer utilizing WebGL API andMapbox Unity SDK.1. INTRODUCTIONproduce the whole building from the bottom to the roof top indetail, completely, and realistically.3D modeling can be generated with various methods, such asUAV photogrammetric and laser scanning methods. Dataacquisition using terrestrial LiDAR is able to achieve highdensity of pointclouds. However, it has low level of pointcloudsdensity in the roof part of tall building due to the fact that TLSwas only placed on the ground. Thus, it will cause incompletebuilding’s pointclouds at the roof part as shown in Figure 1.Figure 2. High density of UAV photogrammetric pointcloudsat the roof part of building.Figure 1. Incomplete building’s pointclouds on the roof part(denoted by red circle) as the result of Terrestrial Lidaracquisition.Fortunately, UAV photogrammetry is able to produce highdensity of pointcluds at the roof part since it carries the camerasensor and captures the object from above. By acquiring imagesthrough UAV, the roof top part can be mapped properly (Figure2). In order to make a complete 3D pointclouds of heritagebuilding, we then merged and integrated the TLS and UAVpointclouds data. The combination of both data are able toFor the data collection, we use Faro Focus X330 and GLS 2000Laser Scanner. UAV DJi Phantom 3 Pro with 12MP cameraresolution was used to acquire the image and produce threedimensional point clouds by using Structure from Motion (SfM)and Multi View Stereo (SfM-MVS) technique using Photoscansoftware.This research mainly has two aims. The first one is to integratethe terrestrial LiDAR and UAV photogrammetry pointcloudsdata using Iterative Closest Point (ICP) algorithms. The next oneis to build interactive displaying of a web 3D model with spatialfunction utilizing WebGL API and Mapbox. For its purpose, amodern and widely accessible game engine technologyThis contribution has been es-XLII-2-W17-39-2019 Authors 2019. CC BY 4.0 License.39

The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLII-2/W17, 20196th International Workshop LowCost 3D – Sensors, Algorithms, Applications, 2–3 December 2019, Strasbourg, France(Unity3D) were used. Unity 3D Game Engine is the highestquality of game engine available in the market, and it is easy touse. 3D game engine can be extended for better visualization,format interoperability and other spatial related to functions orapplications in the near future (Buyuksalih et al., 2017).2. RELATED WORKSThis paper presents an automatic method for integrating Laserscanning and UAV pointclouds data. The roofs are extractedfrom UAV image and the building’s wall are extracted fromterrestrial LiDAR equipment. Although 3D models are useful topreserve the information of heritage building, they will not befully accomplished until it delivers interactively andcommunicates their potential functions to the users.We collected 49 scan stations at KGM object, and 102 scanstations for VFM object with an average scan distance of 25m(Figure 5). The terrestrial LiDAR instrument distributed the laserbeam at a vertical range of 270 and horizontal range of 360 which was able to produce 1 million points per second. The resultof a scan acquisition was a huge number of points in 3D spacewhich represented the object’s shape. Each point had an x, y, zcoordinate and usually a laser reflectance value. Moreover, thepoints had a colour information in the form of RGB values. It wasprovided by a digital camera installed on the scanner system. Forthe UAV data collection, we had more than 600 images with 60mflying altitude and achieved the Ground Sampling Distance 2.18cm.(a)(Laksono and Aditya, 2019) explored the Unity 3D game enginevisualizing large-scale topographic data from mixed sources ofterrestrial laser scanner models and topographic map data. Theinteractive visualization displaying as a game platform allowedthe users to explore the 3D models in LOD3 level. The use ofmodern and widely accessible game engine technology(Unity3D) was also explored by (Albourae et al., 2017) whointegrated pipeline between HBIM, GIS and augmented andvirtual reality (AVR) tools.Data integration are solution to create complete 3D database ofthe complex and tall architectural structure as proposed by(Achille et al., 2015) who merged with three methods; TLS data,close range photogrammetry and aerial survey. The solution forthe 3D modelling of tall structure is integration of laser scannerdata of the internal areas with dense image matching DSM of theexternal facade and roof part-based photogrammetry. Anotherway to align datasets by matching the surface geometry isproposed by (Akca, 2010). The algorithms estimate the Euclidiandistances between surface patches by least squares and minimizethe distance iteratively.(b)3. METHODOLOGY3.1 Data Acquisition and Study AreaIn this study, we collected two architectural heritage building inYogyakarta,.i.e. "Vredeburg Fort Museum (VFM)", a formercolonial fortress built by the Dutch in 1760 and "Kotagede GreatMosque (KGM)", the largest mosque attributed to the kingdomof Mataram, Yogyakarta City, Indonesia.Figure 5. Terrestrial Lidar’s scan station at Kotagede GreatMosque Building (a) and Vredeburg Fort Museum (b).Red circle denotes TLS scan stationThose data were then processed with the suitable algorithms andcompatible software. The workflow and corresponding softwareused in the process were illustrated in Figure 6.Figure 3. Vredeburg Fort Museum BuildingFigure 4. Kotagede Great Mosque BuildingFigure 6. Workflows for data integration and 3DvisualizationThis contribution has been es-XLII-2-W17-39-2019 Authors 2019. CC BY 4.0 License.40

The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLII-2/W17, 20196th International Workshop LowCost 3D – Sensors, Algorithms, Applications, 2–3 December 2019, Strasbourg, France3.2 udsWe used Agisoft Metashape software to generate 3D point cloudsfor both of heritage building. The workflow consists severalprocessing steps. It detect and match the feature points indifferent camera perspectives by using Scale Invariant FeatureTransform (SIFT) algorithms, reconstructing 3D scene structureas well as solving the intrinsic and extrinsic orientationparameters of the camera and generating a sparse point cloudthrough SfM (Smith et al., 2015). We also conducted on-the-jobself-calibration for calculating interior orientation parameter.Finally, we used MVS dense image matching algorithm toproduce 3D dense point cloud. UAV dense pointclouds arereliable, applicable, acceptable accuracy, and almost has equalresolution to TLS techniques.However, the pointclouds computed from SfM and MVSalgorithms were still noisy and sparse. To overcome theseproblems, we applied Random Forest Machine Learning (RFML)to classify and remove non-building object provided by Lidar360-Green Valley Software. There are 3 main steps while usingRFML to remove noisy pointclouds, i.e. creating and defining thetraining data, generating trained model and classifying the data(Green-Valley, 2018). The machine learning classification usesthe trained model for determining individual point classificationsbased on a statistical model of user’s defined feature types.Random Forests were successfully applied to classifypointclouds derived from photogrammetric reconstruction,especially for building object.3.3 TLS Pointclouds RegistrationSince each scan world is defined from the coordinate scannersystem (local coordinate), then it needs to register all of scanworld pointclouds data. Pointclouds registration needs a fixedposition and orientation from the coordinate system scannerwhich appropriate for global coordinate system. Generally,registration techniques are divided into two categories: direct andindirect procedures. Here, we used indirect registration base oncloud registration. It uses the result of overlap (30% - 40%) ofpoint clouds which is also called the Iterative Closed Point (ICP)processing technique. This algorithms is used to find the rigidtransformation between the target point and the reference point,so that the two matching data satisfy the optimal match undersome kind of metric criterion (He et al., 2017). For this purposes,Faro Scene software was used, and what we got was that the RMSerror of registration was 2mm.3.4 Combining and Integrating Pointclouds DataFrom the previous steps, we already had a registered pointcloudsdata from TLS data and UAV imagery. However, both of thesedata were not registered yet. Thus, we integrated them into onereference system in order to produce the entire building as wellas the roof part. For this purpose, we applied the same algorithms,i.e, ICP, to integrate both of pointclouds data. It requires anestimation of the relative position between two data source at theoverlapping area.We set pointclouds from TLS as reference data. Based onoverlapping area, algorithm searches for each point in TLS datawhich is the closest point in the UAV data and uses thecorresponding point pairs to compute a new relative position.This process is repeated iteratively until the relative position ofthe scans converges (Alshawabkeh and Haala, 2004). The ICPalgorithm works well if the pointclouds sources contain relativelylarge areas of continuous surfaces and have sufficient overlap.3.5 Build 3D ModelDifferent types of 3D models can be constructed frompointclouds data, such as primitives model, mesh models orhybrid models. Based on pointlouds data, the 3D model can bereconstructed semi automatically and integrated into BIMsoftware (Macher et al., 2017). In this research, we usedAutodesk Maya, a program to build model geometry into variousprimitive models. The easiest way to start creating complicatedmodels is to begin with 3D primitive shapes. Maya has multiplepre-made shapes that we can use as a starting base. These areknown as Primitives model. These Primitives model includescone, sphere, pyramid, torus, cylinder, wedge and box (Autodesk,2017) as shown in Figure 7.Figure 7. 3D primitive shapesWe built these primitive shape manually base on integratingpointclouds data (Figure 8). In Maya we could integrate imagesfrom other sources for texturing the 3D models. We generated acolor texture base on TLS color pointclouds combined withterrestrial photo captured by digital camera. After texturing hadbeen completed, then it exported the 3D model into FBX formatfor further data processing in Unity3D.Figure 8. Build 3D primitive model by using Autodesk Maya3.6 Game Engine VisualizationGame engines such as Unity3D ( has beenused in various cases for 3D visualization of real-world data(Buyuksalih et al., 2017). The main advantage of Unity3D is thatit can run on multiplatform OS including Windows, Mac, Linuxand also mobile. It is capable of importing other 3D models infbx, sbx and obj formats (Mat et al., 2014). The adde