Skip to content
BY 4.0 license Open Access Published by De Gruyter Open Access March 18, 2021

Towards the Modelling of Veillance based Citizen Profiling using Knowledge Graphs

  • Siraj Munir , Syed Imran Jami EMAIL logo and Shaukat Wasi
From the journal Open Computer Science


In this work we have proposed a model for Citizen Profiling. It uses veillance (Surveillance and Sousveillance) for data acquisition. For representation of Citizen Profile Temporal Knowledge Graph has been used through which we can answer semantic queries. Previously, most of the work lacks representation of Citizen Profile and have used surveillance for data acquisition. Our contribution is towards enriching the data acquisition process by adding sousveillance mechanism and facilitating semantic queries through representation of Citizen Profiles using Temporal Knowledge Graphs. Our proposed solution is storage efficient as we have only stored data logs for Citizen Profiling instead of storing images, audio, and video for profiling purposes. Our proposed system can be extended to Smart City, Smart Traffic Management, Workplace profiling etc. Agent based mechanism can be used for data acquisition where each Citizen has its own agent. Another improvement can be to incorporate a decentralized version of database for maintaining Citizen profile.

1 Introduction

Profiling is the process of inferencing about a person based on known facts [1]. Integration of ubiquitous devices within our life have enabled us to use technological sensory devices and extract different genre of data from it regardless of entities. Entity can be anything such as person, vehicle, transactions, and behavior. Due to enormous growth of population, we need a mechanism to profile citizens from various aspects such as behavior, social interaction, and abiding to the law. Citizen Profiling is a cross-disciplinary area where we observe citizens using various devices including IoT (Internet of Things) based smart devices, sensors, microcontrollers etc. It requires support from different disciplines including lawyers, developers, scholars etc. Democratic states like Pakistan have their own rules and regulations regarding citizen's privacy and security. With the help of multi-disciplinary efforts, we can build a framework that can take care of each aspect of Citizen Profiling [2]. The main problem in this research is to propose a solution for Citizen Profiling. The following sub problems are identified.

  1. To develop, design and integrate sousveilance based system for Data Acquisition.

  2. To model Knowledge Graph which can help in determining semantic queries for citizen profiling.

Veillance is originated from the French word “veiller” which means “to watch” [3]. Veillance is an act of doing vigilance on entities. The use of technology in Citizen Profiling leads to autonomous Citizen Profiling without human intervention. This profiling information can be used by ordinary citizens, industry owners, and defense authorities. Examples include a father querying to know about his son's whereabouts on a particular day, the places he visited, and the location during his college time [2]. The following is the list of queries that Citizen Profiling can resolve:

  1. Which places did XYZ person visit all day?

  2. Which route(s) XYZ person usually takes when going to office?

  3. How many people visited the main gate today at specific instance of time?

  4. How many people attended XYZ seminar/workshop?

  5. How many students were present in the Computing Lab when a computer short-circuit happened?

  6. Which particular locations in the university faced congestion today?

  7. How many faculty members attended today's meeting on ethical issues?

  8. How many ambulances reached on time when certain events occurred?

  9. How many people were present at the opening ceremony of new department?

  10. How many people visited the particular department(s)?

  11. How many and which departments John visited in his last visit to campus?

  12. Which library/department, Tim visits more often?

  13. How many students visited admin office today?

The list is not exhaustive. Many questions related to current and past events are required to be probed while future events can be predicted. Implementation aspects and results extracted from the aforementioned queries have been discussed in the results section. Let us suppose that we come to a situation where government employee attitude with citizens was unacceptable. We have noticed on several occasions that employee behavior was not ethical and the major thing to observe was that these incidents were under surveillance. While in some incidents, officials did not consider taking any action or in some scenarios surveillance clip was not available. This particular situation raises questions on the validation of surveillance systems. Such an issue can be resolved by integrating sousveillance with surveillance. Surveillance has already proved its potential implications in different areas like computing and social sciences [2, 3]. Using surveillance and sousveillance collectively would help in resolution of different queries that are still complex to solve by surveillance systems alone. In Figure 1 we have presented a detailed taxonomy of veillance that differentiates types of veillance and their sub-types. However, there are some major problem with surveillance which we also reported in [4]:

  1. Surveillance is a special type of veillance that is about/around a fixed point.

  2. What will happen if the surveillance system was updating while certain events happened?

  3. What will happen if any high-profile person uses resources and deletes surveillance data from the system?

  4. What can we do in a situation where the server was down when the particular terrorism event happened?

Figure 1 Taxonomy of Veillance [4, 6]
Figure 1

Taxonomy of Veillance [4, 6]

These are some of the potential questions (queries) though the list is not exhaustive. One complete and potentially valid solution towards the highlighted queries is integration of sousveillance with surveillance. We are living in the age where we have ubiquitous technology everywhere around us. Using technology with the right inception and perception can eventually sort out many complex situations that occur in our daily life.

In this research, we have proposed a novel model for Citizen Profiling which would be capable to resolve intelligent queries as mentioned above. We have used various sousveillance and surveillance devices to provide the information that is represented as a Knowledge Graph for inferring and matching relationships among the entities. The major contributions of this work are: (i) Autonomous Citizen Profiling using sousveillance integrated with surveillance, (ii) Embedding temporal profiling information with Knowledge Graphs, and (iii) Integrated temporal and dynamic features in Knowledge Graphs. For the proof of concept, we have used a campus environment which would be extendable to a metropolitan level. The proposed techniques can also add contribution to Industry 4.0, Smart City, and traffic management system. This paper is organized into five sections. The first section briefly discusses what is Citizen Profiling and provides insights on different aspects of Citizen Profiling. The next section provides a detailed related literature work in correspondence with this research. The third section discusses the proposed methodology, including Data modelling and Semantic layer. The fourth section covers up results and discussion which we have achieved so far. The last section provides concluding remarks on the proposed work.

2 Background and related work

Citizen Profiling is a cross disciplinary area involving Computer Science, Social Science, and Law. Recent state of the art technologies like IoT (Internet of Things), Collective Intelligence, and Crowdsourcing are providing cutting edge solutions to solve complex problems in a distributed setting. The next subsections provide insight on major components and related them to existing work.

2.1 Veillance Layer

Steve Mann [5, 6] has introduced various forms of veillance. This work focused on using surveillance and sousveillance for data acquisition in Citizen Profiling setting. Recent works show a wide range of problems that are supposed to be solved through surveillance and computer vision [7,8,9,10,11,12,13,14]. Studies in Surveillance have focused more on computer vision in which the datasets contain a collection of images that can create storage overheads. In [7] authors discussed video surveillance, which were employed across Britain by BSIA (British Security Industry Association). The authors proposed a novel approach for dealing with hierarchical architecture using FVSA (Fused Video Surveillance Architecture) approach that includes integration of multilayer abstraction towards Internet of Things (IOT) based systems. The authors in [9] discussed behavioral aspects of employees through surveillance and computer-mediated technique within the organization. The survey was done on full-time employees to ensure trust, and privacy and their commitment with the organization.

The study in [15] experimented the effectiveness of ‘Surveillance of Things’ in contrast to the Internet of Things. For implementation of experimental setup, authors employed Sigfox which offers wireless framework for IoT oriented networks. Moreover, the authors also looked forward into a question of surveillance of people connected to SoT (surveillance of things). In [16] authors discussed implication of Big Data for surveillance oriented systems, especially for the scenarios of digital citizenship and contemporary democracy. In the literature authors shown the critical analysis of five works with the suggestion on how one can use ambivalent digital citizen mechanism that can target every citizen. The work in [17] discusses RFID usage for the object detection that can also be integrated with surveillance system(s). In [18], the authors introduced a novel open-source database which includes videos that can be further used for object-entity tracking tasks, object-entity recognition tasks and entity-object re-identification tasks. The proposed dataset also contains a bunch of high-resolution pictures that can also be employed for experimental setups for validating different algorithms. The literature presented in [19] acknowledges some prior and recently proposed approaches for object detection in variable environments. Proposed literature also discusses several areas where object detection can help. Liu and Young in [20] briefly discuss the implication of social media and surveillance data from different aspects like behavioral analysis and medical news analysis. In [10] authors discuss challenges and practical implications of surveillance for moving objects and in response to these challenges, authors proposed several approaches that can aid in resolution of the highlighted challenges. Ohn-Bar et al. in [5] highlighted surveillance and its applications for CV (Computer Vision) in ‘assist drivers’ for safety critical events. Further, the authors also validated the proposed approach over the different complex datasets. Saemi et al. [11] proposed to bridge the analysis between classic and recent object-entity detection techniques. The article also details that in practical environments, we have bulk of the videos (surveillance) but due to variability in detection of objects we face several problems which lead to the complexity of tasks. The proposed article introduced a novel framework which uses BOW (Bag-of-word) and classification collectively to resolve the highlighted problem effectively. Experiments show that the proposed approach successfully achieves the top accuracy of 92% on LOST (Long-term Observation of Scenes (with Tracks)) data collection. The literature presented in [12] highlighted some optimized and cost-efficient approaches for object detection dynamically. Tsakanikas and Dagiuklas [13] discuss the current and future trends in the domain of surveillance systems. The authors discussed several techniques that can be used for entity detection, picture enhancement, entity tracking, and entity recognition. Authors also stressed upon the employment of latest architectural solutions like cloud computing, fog computing or mobile-edge computing. Lastly, authors wrapped up literature with the expectation from surveillance systems in the future, such as Augmented Reality for surveillance. In [14] authors stressed upon multi-cue entity detection for the variable natured environment. For completion of desired task, authors proposed modularity based single modal or multimodal approach. The literature in [21] discusses an intelligent approach for crowd segmentation/classification using surveillance videos. For crowd segmentation authors experimented with historical statistical analysis and showed that the proposed technique was efficient enough to be used as surveillance system.

Sousveillance was introduced by Steve Mann [22]; it can be explained as inverse-surveillance or surveillance on-demand. Sousveillance can also be described as monitoring of particular entity for instance car, student, or children using wearable computing devices. Research trends show that area of sousveillance has not received attention in computing research community in recent years as discussed at the end of the previous section.

Surveillance based approaches still have received attention from the research community to improve surveillance techniques [11, 14]. There are several use cases for integrated usage of surveillance and Sousveillance. Trends show that sousveillance data aids in scenarios where surveillance fails to provide solutions.

2.2 Semantic Layer

The latest trends in Knowledge Graphs show that it is extensively used in modeling of domains for semantic integration. The articles presented in [23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56] show how the Knowledge Graph is used for semantic integration across different domains using NLP, and ML.

Literature in [23] proposes a Knowledge Base (KB) for cyber security. The authors also present deduced rules based on quintuple model. For enhancement of results, authors used Machine Learning (ML) approach on top of ontologies to predict rules using path-ranking algorithm and Conditional Random Field (CRF). Finally, the authors present ontologies that paced up implementations on Cyber Security Knowledge Graph (KG). The work in [28] implemented Knowledge Graph in education. Data collection was made through heterogeneous sources. For extraction of relations and their mapping authors used neural sequence labelling. For proof of concept, authors implemented Knowledge Graph for mathematics through proposed algorithms and achieved F1 score of 70% in relation identification tasks. The literature in [29] proposed Knowledge Graphs for mitigating problems relating to intangible cultural heritage in China. As part of preprocessing, authors collected data from different sources. The ontologies are designed and implemented using Natural Language Processing (NLP) technique. The generated ontologies were refined as per specifications by involving domain experts and engineers. Afterwards, they proposed Knowledge Graphs based on the implemented ontologies.

The work in [34] proposed a workflow for information extraction of Chinese Geoscience documents. This work-flow entails a pipeline from raw text to Knowledge Graph which can ultimately boost up progress of NLP and Knowledge Graph for Geoscience. The literature in [42] proposed an embedding approach that adapts minimization of loss function using Path-specific Margin-based Loss Function for Knowledge Graph (PaSKoGE). It further compared the proposed approach whose results outperformed earlier state-of-the-art approaches. Bean et al. [41] introduced a novel Knowledge Graph that uses a machine Learning (ML) algorithm to predict adverse reactions of known drugs with a remarkable accuracy of 92%. The proposed algorithm outperformed various ML algorithm which includes Logistic Regression, Decision Trees and Support Vector Machines. The article in [44] addressed two major problems that occurred in incomplete Knowledge Graphs when integrated with Question Answering (QA) systems. These problems include false prediction on incomplete Knowledge Graph, that can lead to misleading and illegitimate prediction; also false negative values that can lead to misleading updates to correct/legitimate values. To resolve the aforementioned issues, authors proposed an enhanced model that implements Reinforcement Learning (RL). For the first problem, the authors proposed a RL policy based on one-hop embedding. One-hop embedding is an effective fuzzy logic based approach that uses fuzzy truth for embedding triples. This methodology effectively reduces false negatives. For the second problem, the authors proposed an RL policy that enforces agents to explore complete diversified paths. Trails showed better performance on benchmark datasets than simple embedding based models. The literature in [45] proposed a model for evaluation of Knowledge Graph embedding. The proposed model encompasses the well-known and effective ML technique Ensemble Learning. Hence the proposed model enhanced the previous Knowledge Graph completion embedding techniques.

In [49] an enhanced approach is proposed for formulating Monte Carlo Tree Search algorithm for semantic navigation in Knowledge Graphs. However, the proposed methodology did not perform well for large data sizes. In [51], the article proposed an entity set expansion problem using path-semantic features. For the extraction of these features, authors used Knowledge Graphs. The proposed implementation was capable to find semantic features from seed entity which was further used to extract entities. The literature also proposed probabilistic models for entity ranking. Authors claimed better results in contrast to prior state-of-the-art techniques. The literature in [52] proposed an effective framework for searching on RDF using Knowledge Graphs. This methodology used triples powered up with keywords which ultimately wipe out complexity and increases efficiency of query results. Furthermore, the proposed methodology also used statistical machine translation backed with automated query relaxation. Moreover, authors also proposed a diverse version for searching over RDF that is maximally marginal relevance. Sawant et al. [53] proposed an unconventional approach named AQQUCN for interpretation and extraction of queries. Implementation was done using multiple Convolutional Neural Network (CNN) in contrast with latent variables. The proposed approach can deal with normal questions to complex structure questions such as keyword sequence. Results show 16–18% improvement in mean average precision.

The work in [54] proposed an approach named TransCoRe (Translation-based Method via Modeling the Correlation of Relations). The proposed methodology uses embedded relation matrix which was split into low dimensional matrices. Through these metrics, the model learns embedding entities and their relations. As a testbed, the authors used WordNet and Freebase that are two renowned collections of textual facts. The paper claimed considerable results for link prediction and triple classification tasks of Knowledge Graphs. Literature in [55], proposed a novel Knowledge Graph approach for indoor scene design. The authors implemented entity-relation model for factual representation. Furthermore, the authors proposed a supervised algorithm for extraction of Knowledge Graphs by using structural and parametric learning. Song and Park [56] proposed a Knowledge Graph enrichment scheme, based on translation-based embedding. For the purpose of enrichment of triples, the authors implemented a regularization scheme through which continual/online learning was introduced into Knowledge Graph space. The proposed approach excellently incorporates newer triples into Knowledge Graph to preserve formal and current embedding schemes.

Research trends show that most of the work has been done in the areas of Machine Learning and Natural Language Processing for profiling. This work will be the first attempt towards a representation of profiling in Knowledge Graphs. The next section discusses the methodology adopted in this research work.

3 Methodology

This section discusses our proposed methodology for ‘Veillance’ based citizen profiling through Knowledge Graphs. Our proposed framework is depicted in Figure 2. It is implemented using a layered approach. Therefore, the methodology is divided into two sub-sections namely data modelling for profiling data and semantic modelling. Detailed discussion on each aspect is done in the respective sub-section. The next sub-section discusses how we modelled data for profiling.

Figure 2 Citizen Profiling Framework [62]
Figure 2

Citizen Profiling Framework [62]

3.1 Data Modelling for profiling data

Data modelling helps out in determining criticalities when designing any system. Major approaches include procedural, logic oriented, conceptual or physical. In this work we developed facial recognition module to locate the person that can help in determining the object's key attributes used in profiling. This include the allocated identity, current time of event, and location that are identified as a tuple (ID, timestamp, and location). This module is implemented using Python in which our algorithm locates a person and then instead of transferring multimedia data it transfers the logs to the NoSQL data source thus it reduces the network and storage cost. For the representation of data, we employed NoSQL oriented architecture which ultimately makes it more efficient and effective. Lastly, for archiving logs, we used (key, value) paired data model which is dumped into a centralized version of MongoDB. The archive is used to keep track of profiling information that is initially uniform and finite in nature. However, it can increase exponentially with the passage of time due to the movement of citizens in an environment. Therefore, the collected logs are stored in NoSQL database. Further, this NoSQL database is used to represent data as Big Data. Profiling data naturally satisfies the properties of Big Data also known as 3V's that is velocity, variety, and volume [57]. The subsequent subsection will discuss the graph model for profiling task.

Figure 3 represents the data acquisition pipeline for Knowledge Graphs. For data acquisition, we extracted curated data from multiple sources including Surveillance Camera and Sousveillance devices as extensions. For the purpose of storage and further processing, we used two different tools; (i) MongoDB that is an open source NoSQL document-oriented database, and (ii) Neo4j that is a top notch graph database. The MongoDB is used for maintaining profile logs that were collected from surveillance sources. The Neo4j is used for the implementation and modelling of Knowledge Graphs. A web portal is developed for querying and inferencing of profiling data to answer semantic profiling questions. The same module can be extended for vehicle profiling in smart cities. The preprocessing mentioned in Figure 2 requires some data cleansing before we extract it for generating semantic information. This step will help in resolving the redundant and inconsistent data of profiling. Finally, this preprocessed data will become input for generating Knowledge Graphs.

Figure 3 Data acquisition pipeline for Knowledge Graph [62]
Figure 3

Data acquisition pipeline for Knowledge Graph [62]

Table 1 represents the Knowledge Graph specification. The proposed Knowledge Graph has 136 nodes (including persons and locations), 4 property labels (including Name, Location, Person and Switch), 3762 mostly temporal connections and 10 relationships (including Knows, Located_at, Connected_with, Study_at, ResearchAssitant_at, and Supervisor_of). The next sub-section discusses the modelling and representation perspective of Semantic Information within Knowledge Graphs.

Table 1

Knowledge Graph Specification

Nodes 136

Labels 4
Relationships 3762 (Mostly temporal)
Relationship Types 10

3.2 Semantic Modelling

For semantical integration of profiling Knowledge Graphs, we need to integrate temporal information which is the requirement in the current scenario to represent real time information about citizens. The major issue comes in maintaining temporal information because of its complexity due to dynamic and unusual behavior of citizen(s). Another issue is time span of periodic updates to maintain accuracy in the requirement of soft real time in profiling information. To deal with the aforementioned problems we used a similar approach as the one presented in [58, 59]. To deal with Semantic Information we can either model it in the form of Knowledge Graph [60] or Ontology [61]. The good part is both approaches follow the structure of triples, that is: Subject, Predicate, Object as shown in Table 3. However, in our case we also need timestamped information, so we added a new attribute as Timestamp which results in the extended triples, that is: Subject, Predicate, Object, Timestamp. Table 2 and 3 show an example of aforementioned proposed schemes which are also proposed by Garcia-Duran et al. [59].

Table 2

Dataset of Knowledge Graph

Person Locations Relationships Timestamp
Siraj Munir Parking Connected_with 2019-05-19 08:00
Asim Wagan President Office Located_at 2019-05-19 01:30
Imran Jami Canopy Located_at 2019-05-19 09:45
Shaukat Wasi HoD CS Connected_with+ Located_at 2019-05-19 10:53
Zainab HR Office Located_at 2019-05-19 07:44
Asad Simulation Lab Connected_with 2019-05-20 11:53
Tafseer Ahmed Khan Programming Lab Located_at 2019-05-19 10:20
Umair Lakahani Engineering Lab Located_at 2019-05-19 10:10
Hamza Gaya Library Located_at 2019-05-19 04:53
Jahanzaib Parking Connected_with 2019-05-19 05:45
Table 3

Examples for maintaining temporal information in Knowledge Graph

Information Relationship
(Siraj, ResearchInternee_at, Maju)

(Siraj, Connected_with, Maju, 2019-05-19 01:13)
ResearchInternee_at (Connected_with, Timestamp)

In this work we have implemented the data model which is shown in Figure 4. In Table 3 we have shown some examples of how we have maintained tuple attributes and temporal knowledge in our Knowledge Graph. The Citizen entity includes students, and faculty members while location shows the deployed placement of surveillance camera. Table 2 shows the information kept in our Knowledge Graph for Citizens Profiling. As a proof of concept, we have modelled several interesting queries about citizen(s) which can be answered by the proposed Knowledge Graph in an autonomous way. The next section will discuss the results.

Figure 4 Graph Data Model for Tracking Citizen profile
Figure 4

Graph Data Model for Tracking Citizen profile

4 Results and Discussion

The major contribution of this work is that we have integrated semantic profiling information in the Knowledge Graph. To ensure our proposed model works well we piloted with two different datasets; (i) Citizen Profiling on campus environment, and (ii) Vehicle profiling data. This section is dedicated for discussing results that were produced by implementing the proposed data model on a large-scale extrapolated data collection which will ultimately give us a proof of concept. Here Figure 5 shows our complete Knowledge Graph that was generated from the initial dataset. The proposed Temporal Knowledge Graph can answer all semantic queries that were mentioned in the introduction section of this paper. Figure 6 depicts the specific knowledge/zoomed in version about entities in Knowledge Graph.

Figure 5 Complete Knowledge Graph
Figure 5

Complete Knowledge Graph

Figure 6 Zoomed in Knowledge Graph
Figure 6

Zoomed in Knowledge Graph

Inferencing is a way through which we can devise the missing blocks from data. In many real-world scenarios we have incomplete information which can lead towards semantic failure. Using inferencing we devise rules that can help us to find missing semantic relations in data. In transitive nature of Knowledge Graphs, we may have indirect relationship between the entities that can be identified using inferencing. Similarly, in the case of Citizen Profiling we may have a lot of missing information which can mislead us towards wrong interpretations. By using inferencing, we can find the missing blocks and make our system robust. Figure 7 shows inferencing query results obtained from our profiling Knowledge Graph.

Figure 7 Query result of inferencing
Figure 7

Query result of inferencing

After piloting queries on Knowledge Graphs, we get an intuition that we can also use the proposed model at metropolitan scale for profiling purposes by incorporating a similar class of data. This would lead to Big Data. By using various means like geolocation, mobile tracking, tracking through physical cameras we can keep track of citizens.

5 Conclusion and Future Work

Citizen Profiling is a growing area based on intelligence merged with surveillance and sousveillance. Knowledge of multiple domains need to be incorporated in the system for effective tracking of entities. Semantic Web has been used to represent the information where domain information is incorporated using Knowledge Graphs. Cypher query language is used for information extraction. For prediction and detection of meaningful patterns, inferencing has been done on tracked data. Query engine can have an intelligent agent that can take queries and retrieve related records by traversing graphs. Another problem related to query engine is that it is rule based. As future work we can empower query engine by integrating NLP based techniques including keyword-based search and Information extraction-based search.

Making a user's private data open for world is indeed violation of democratic laws and policies. Research trends show that profiling does not give warrant for user data protection and do not have any effective legal and technological framework. Therefore, the area requires research in cross disciplinary issues where we need the integration of researchers from the areas of data science, artificial intelligence, psychology and law. This will help in understanding the actor's behavior within environments in an autonomous and pervasive setting. Features like autonomous and pervasiveness may raise the issue of legal hurdles for which we need to propose laws and regulations to protect illegitimate, illegal or negative use of profiling data especially in third world countries. Domain experts from law and regulatory bodies will be required to establish and incorporate privacy and data protection laws which can help in answering questions mentioned in the previous sections. Many problems related to criminal tracking, car tracking, terrorism, traffic monitoring, and disaster management can be solved using this area. Furthermore, there can be several improvements that can be made in this work. This includes use of decentralized storage mechanism as sousveillance recommends use of distributed system so that we can have data on multiple nodes in storage. Moreover, Multi-Agent and Adaptive systems can be incorporated for making profiling mechanism more robust. By using Multi-Agent systems, we can allocate separate agents for each task or event related to citizens. This leads to another research areas including Agent Society, Negotiation, and Interaction. By using Adaptive systems, we can make context aware interfaces which can show refined results based on roles to preserve privacy. For instance, if a person is a father then Adaptive system should show data related to father. Collectively Multi-Agent and Adaptive system will enhance the overall performance of profiling systems. We can also make specialized ontologies for individual purposes and on top of them implement Knowledge Graphs so that we can process intelligent queries over them effectively. Effective implementation of Citizen Profiling would also add value to Industry 4.0 revolution.


[1] IGI-Global,, 2019Search in Google Scholar

[2] Hildebrandt, M., Profiling and the Identity of the European Citizen, In: Hildebrandt, M., and Gutwirth, S. (Eds.): Profiling the European Citizen: Cross-Disciplinary Perspectives, (Springer Netherlands, 2008), 303–34310.1007/978-1-4020-6914-7_15Search in Google Scholar

[3] Mann, S., Veilance and reciprocal transparency: Surveillance versus sousveillance, AR glass, lifeglogging, and wearable computing, IEEE International Symposium on Technology and Society (ISTAS): Social Implications of Wearable Computing and Augmediated Reality in Everyday Life, Toronto, ON, 2013, 1–1210.1109/ISTAS.2013.6613094Search in Google Scholar

[4] Munir, S., and Jami, S., Research Trends in Surveillance through Sousveillance, International Journal of Advanced Computer Science and Applications, 2019, 10(12), 433–43710.14569/IJACSA.2019.0101258Search in Google Scholar

[5] Ohn-Bar, E., Tawari, A., Martin, S., and Trivedi, M.M.: On surveillance for safety critical events: In-vehicle video networks for predictive driver assistance systems, Computer Vision and Image Understanding, 2015, 134, 130–14010.1016/j.cviu.2014.10.003Search in Google Scholar

[6] Mann S., Surveillance (Oversight), Sousveillance (Undersight), and Metaveillance (Seeing Sight Itself), IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2016, 1408–1417.10.1109/CVPRW.2016.177Search in Google Scholar

[7] Ajiboye, S.O., Birch, P., Chatwin, C., and Young, R., Hierarchical video surveillance architecture: a chassis for video big data analytics and exploration, SPIE/IS&T Electronic Imaging, 2015, 10.10.1117/12.2083937Search in Google Scholar

[8] Chahyati, D., Fanany, M.I., and Arymurthy, A.M., Man woman detection in surveillance images, 5th International Conference on Information and Communication Technology (ICoIC7), 2017, 1–4.10.1109/ICoICT.2017.8074682Search in Google Scholar

[9] Chory, R.M., Vela, L.E., and Avtgis, T.A.: Organizational Surveillance of Computer-Mediated Workplace Communication: Employee Privacy Concerns and Responses, Employee Responsibilities and Rights Journal, 2016, 28(1), 23–4310.1007/s10672-015-9267-4Search in Google Scholar

[10] Mishra, P.K., and Saroha, G.P, A study on video surveillance system for object detection and tracking, 3rd International Conference on Computing for Sustainable Global Development (INDIA-Com), 2016, 221–226.Search in Google Scholar

[11] Saemi, M.M., See, J., and Tan, S., Lost and found: Identifying objects in long-term surveillance videos, 2015 IEEE International Conference on Signal and Image Processing Applications (ICSIPA), 2015, 99–104.10.1109/ICSIPA.2015.7412171Search in Google Scholar

[12] Sajjanar, S., Mankani, S.K., Dongrekar, P.R., Kumar, N.S., Mohana, and Aradhya, H.V.R., Implementation of real time moving object detection and tracking on FPGA for video surveillance applications, IEEE Distributed Computing, VLSI, Electrical Circuits and Robotics (DISCOVER), 2016, 289–295.10.1109/DISCOVER.2016.7806248Search in Google Scholar

[13] Tsakanikas, V., Dagiuklas, T., Video surveillance systems-current status and future trends, Computers & Electrical Engineering, 2018, 70(May 2018), 736–753.10.1016/j.compeleceng.2017.11.011Search in Google Scholar

[14] Walia, G.S., and Kapoor, R., Robust object tracking based upon adaptive multi-cue integration for video surveillance, Multimedia Tools and Applications, 2016, 75(23), 15821–15847.10.1007/s11042-015-2890-0Search in Google Scholar

[15] Doucek, P., Pavlicek, A., and Luc, L., Internet of Things or Surveillance of Things?, In: Research and Practical Issues of Enterprise Information Systems, 2018, Springer, 45–55.10.1007/978-3-319-94845-4_5Search in Google Scholar

[16] Duncan J., Uncertainty and Desire: Big Data Surveillance and Digital Citizenship, The iJournal: Graduate Student Journal of the Faculty of Information, 2018, 3(3).Search in Google Scholar

[17] Fescioglu-Unver, N., Choi, S.H., Sheen, D., and Kumara, S., RFID in production and service systems: Technology, applications and issues, Information Systems Frontiers, 2015, 17 (6), 1369–138010.1007/s10796-014-9518-1Search in Google Scholar

[18] Fularz, M., Kraft, M., Schmidt, A., and Niechciał, J.: The PUT Surveillance Database, In Choras R. (Ed.): Image Processing and Communications Challenges, 2016, Springer, 73–79.10.1007/978-3-319-23814-2_9Search in Google Scholar

[19] Kulchandani, J.S., and Dangarwala, K.J. Moving object detection: Review of recent research trends, International Conference on Pervasive Computing (ICPC), Pune, 2015, 1–510.1109/PERVASIVE.2015.7087138Search in Google Scholar

[20] Liu, S., and Young, S.D. A survey of social media data analysis for physical activity surveillance, Journal of Forensic and Legal Medicine, 2018, 57, 33–3610.1016/j.jflm.2016.10.019Search in Google Scholar PubMed PubMed Central

[21] Yang, S., Yang, H., Li, J., and Zhu, J., An Effective Crowd Property Analysis System for Video Surveillance Application, Digital TV and Wireless Multimedia Communication Springer, Singapore, 2017, 115–12710.1007/978-981-10-4211-9_12Search in Google Scholar

[22] Mann, S., Sousveillance: inverse surveillance in multimedia imaging, In: Proceedings of the 12th annual ACM international conference on Multimedia, 2004, ACM, 620–62710.1145/1027527.1027673Search in Google Scholar

[23] Jia, Y., Qi, Y., Shang, H., Jiang, R., and Li, A., A Practical Approach to Constructing a Knowledge Graph for Cybersecurity, Engineering, 2018, 4(1), 53–6010.1016/j.eng.2018.01.004Search in Google Scholar

[24] Oldman, D., and Tanase, D., Reshaping the Knowledge Graph by Connecting Researchers, Data and Practices in ResearchSpace, The Semantic Web – ISWC 2018, 2018, Springer, 325–34010.1007/978-3-030-00668-6_20Search in Google Scholar

[25] Pujara, J., and Singh, S., Mining Knowledge Graphs From Text., In: Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, 2018, ACM, 789–790.10.1145/3159652.3162011Search in Google Scholar

[26] Chun, S., Jin, X., Seo, S., Lee, K., Shin, Y., and Lee, I., Knowledge Graph Modeling for Semantic Integration of Energy Services, IEEE International Conference on Big Data and Smart Computing (BigComp), 2018, IEEE, 732–73510.1109/BigComp.2018.00138Search in Google Scholar

[27] Wang, R., Yan, Y., Wang, J., Jia, Y., Zhang, Y., Zhang, W., and Wang, X., AceKG: A Large-scale Knowledge Graph for Academic Data Mining. In: Proceedings of the 27th ACM International Conference on Information and Knowledge Management, 2018, ACM, 1487–1490.Search in Google Scholar

[28] Chen, P., Lu, Y., Zheng, V.W., Chen, X., and Yang, B., KnowEdu: A System to Construct Knowledge Graph for Education, IEEE Access, 2018, 6, 31553–3156310.1109/ACCESS.2018.2839607Search in Google Scholar

[29] Dou, J., Qin, J., Jin, Z., and Li, Z.: Knowledge graph based on domain ontology and natural language processing technology for Chinese intangible cultural heritage, Journal of Visual Languages & Computing, 2018, 48, 19–2810.1016/j.jvlc.2018.06.005Search in Google Scholar

[30] Silva, V., Freitas, A., and Handschuh, S., Building a Knowledge Graph from Natural Language Definitions for Text Entailment Recognition, Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), 2018, European Language Resources Association.Search in Google Scholar

[31] Fathalla, S., and Lange, C., EVENTSKG: A Knowledge Graph Representation for Top-Prestigious Computer Science Events Metadata, In: Computational Collective Intelligence, 2018, Springer, 53–63.10.1007/978-3-319-98443-8_6Search in Google Scholar

[32] Patel, H., Paraskevopoulos, P., and Renz, M.: GeoTeGra: A System for the Creation of Knowledge Graph Based on Social Network Data with Geographical and Temporal Information, in IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM), 2018, IEEE/ACM, 617–62010.1109/ASONAM.2018.8508674Search in Google Scholar

[33] Cheng, B., Zhang, Y., Cai, D., Qiu, W., and Shi, D.: Construction of traditional Chinese medicine Knowledge Graph using Data Mining and Expert Knowledge, International Conference on Network Infrastructure and Digital Content (IC-NIDC), 2018, IEEE, 209–21310.1109/ICNIDC.2018.8525665Search in Google Scholar

[34] Wang, C., Ma, X., Chen, J., and Chen, J., Information extraction and knowledge graph construction from geoscience literature, Computers & Geosciences, 2018, 112, 112–12010.1016/j.cageo.2017.12.007Search in Google Scholar

[35] Kartsaklis, D., Pilevar, M.T., and Collier, N., Mapping Text to Knowledge Graph Entities using Multi-Sense LSTMs, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018, ACL, 1959–197010.18653/v1/D18-1221Search in Google Scholar

[36] Luan, Y., He, L., Ostendorf, M., and Hajishirzi, H., Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Construction, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018, ACL, 3219–323210.18653/v1/D18-1360Search in Google Scholar

[37] Guo, L., Zhang, Q., Ge, W., Hu, W., Qu, Y., DSKG: A Deep Sequential Model for Knowledge Graph Completion, In: Zhao, J., Harmelen, F., Tang, J., Han, X., Wang, Q., Li, X. (eds), Knowledge Graph and Semantic Computing. Knowledge Computing and Language Understanding. Communications in Computer and Information Science, 2019, Springer, 957, 65–77.10.1007/978-981-13-3146-6_6Search in Google Scholar

[38] Kampffmeyer, M., Chen, Y., Liang, X., Wang, H., Zhang, Y., and P. Xing, E., Rethinking Knowledge Graph Propagation for Zero-Shot Learning, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, IEEE, 11479–11488,10.1109/CVPR.2019.01175Search in Google Scholar

[39] Palumbo, E., Rizzo, G., Troncy, R., Baralis, E., Osella, M., and Ferro, E., Knowledge Graph Embeddings with node2vec for Item Recommendation, The Semantic Web – ISWC 2018, 2018, Springer, 117–12010.1007/978-3-319-98192-5_22Search in Google Scholar

[40] Wang, Z., Lv, Q., Lan, X., and Zhang, Y.: Cross-lingual Knowledge Graph Alignment via Graph Convolutional Networks, in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018, ACL, 349–35710.18653/v1/D18-1032Search in Google Scholar

[41] Bean, D.M., Wu, H., Dzahini, O., Broadbent, M., Stewart, R., and Dobson, R.J.B., Knowledge graph prediction of unknown adverse drug reactions and validation in electronic health records, Scientific Reports, 2018, 7(1), 428410.1038/s41598-018-22521-4Search in Google Scholar PubMed PubMed Central

[42] Jia, Y., Wang, Y., Jin, X., and Cheng, X., Path-specific knowledge graph embedding, Knowledge-Based Systems, 2018, 151, 37–4410.1016/j.knosys.2018.03.020Search in Google Scholar

[43] Annervaz, K.M., Chowdhury, S.B.R., and Dukkipati, A., Learning beyond datasets: Knowledge Graph Augmented Neural Networks for Natural language Processing, In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2018, 1, 313–322Search in Google Scholar

[44] Lin, V., Socher, R., and Xiong, C.: Multi-Hop Knowledge Graph Reasoning with Reward Shaping, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018, ACL, 3219–323210.18653/v1/D18-1362Search in Google Scholar

[45] Meilicke, C., Fink, M., Wang, Y., Ruflnelli, D., Gemulla, R., and Stuckenschmidt, H.: Fine-Grained Evaluation of Rule- and Embedding-Based Systems for Knowledge Graph Completion’, In: The Semantic Web – ISWC 2018, 2018, Springer, 3–20.10.1007/978-3-030-00671-6_1Search in Google Scholar

[46] Alam, M., Gangemi, A., Presutti, V., and Recupero, D. R., Semantic Role Labeling for Knowledge Graph Extraction from Text, 2018, arXiv e-prints.Search in Google Scholar

[47] Hong, S., Park, N., Chakraborty, T., Kang, H., and Kwon, S., ‘PAGE: Answering Graph Pattern Queries via Knowledge Graph Embedding’, In: Chin F., Chen C., Khan L., Lee K., Zhang L.J. (Eds), Big Data – BigData 2018, Springer, Lecture Notes in Computer Science, 10968, 87–9910.1007/978-3-319-94301-5_7Search in Google Scholar

[48] Asadifar, S., Kahani, M., Shekarpour, S., HCqa, Hybrid and Complex Question Answering on Textual Corpus and Knowledge Graph, 2018, arXiv e-prints.Search in Google Scholar

[49] He, L., Shao, B., Xiao, Y., Li, Y., Liu, T., Chen, E., and Xia, H., Neurally-Guided Semantic Navigation in Knowledge Graph, IEEE Transactions on Big Data, 2018, 1–110.1109/TBDATA.2018.2805363Search in Google Scholar

[50] Wu, P., Zhou, Q., Lei, Z., Qiu, W., and Li, X., Template Oriented Text Summarization via Knowledge Graph, in 2018 International Conference on Audio, Language and Image Processing (ICALIP), 2018, 79–83.10.1109/ICALIP.2018.8455241Search in Google Scholar

[51] Chen, J., Chen, Y., Zhang, X., Du, X., Wang, K., and Wen, J.-R., Entity set expansion with semantic features of knowledge graphs, Journal of Web Semantics, 2018, 52(53), 33–4410.1016/j.websem.2018.09.001Search in Google Scholar

[52] Arnaout, H., and Elbassuoni, S., Effective searching of RDF knowledge graphs, Journal of Web Semantics, 2018, 48, 66–8410.1016/j.websem.2017.12.001Search in Google Scholar

[53] Sawant, U., Chakrabarti, S., and Ramakrishnan, G., Open-domain question answering using a knowledge graph and web corpus, ACM SIGWEB Newsletter, 2018, 1–810.1145/3183639.3183643Search in Google Scholar

[54] Zhu, J. Z., Jia, Y. T., Xu, J., Qiao, J. Z., and Cheng, X. Q., Modeling the Correlations of Relations for Knowledge Graph Embedding, Journal of Computer Science and Technology, 2018, 33 (2), 323–33410.1007/s11390-018-1821-8Search in Google Scholar

[55] Liang, Y., Xu, F., Zhang, S.-H., Lai, Y.-K., and Mu, T., Knowledge graph construction with structure and parameter learning for indoor scene design, Computational Visual Media, 2018, 4 (2), 123–13710.1007/s41095-018-0110-3Search in Google Scholar

[56] Song, H., and Park, S., Enriching Translation-Based Knowledge Graph Embeddings Through Continual Learning, IEEE Access, 2018, 6, 60489–6049710.1109/ACCESS.2018.2874656Search in Google Scholar

[57] Jagadish, H. V., Gehrke, J., Labrinidis, A., Papakonstantinou, Y., Patel, J. M., Ramakrishnan, R., Shahabi, C., Big data and its technical challenges, Communications of the ACM, 2014, ACM, 57(7), 86–94.10.1145/2611567Search in Google Scholar

[58] Gottschalk, S., and Demidova, E., EventKG: A Multilingual Event-Centric Temporal Knowledge Graph, In: Proceedings of the 15th Extended Semantic Web Conference (ESWC 2018), 201810.1007/978-3-319-93417-4_18Search in Google Scholar

[59] Garcia-Duran, A., Dumancic, S., Niepert, M.: Learning Sequence Encoders for Temporal Knowledge Graph Completion’, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018, ACL, 4816–482110.18653/v1/D18-1516Search in Google Scholar

[60] Wang, Q., Mao, Z., Wang, B., Guo, L., Knowledge graph embedding: A survey of approaches and applications, IEEE Transactions on Knowledge and Data Engineering, 2017, 29(12), 2724–2743.10.1109/TKDE.2017.2754499Search in Google Scholar

[61] Guarino, N., Oberle, D., Staab, S., What is an ontology?, In: Staab, S., Studer, R., (Eds), Handbook on ontologies, Springer, 1–17.10.1007/978-3-540-92673-3_0Search in Google Scholar

[62] Jami, I., Wasi, S., and Munir, S., Knowledge Graph based Semantic Modeling for Profiling in Industry 4.0, International Journal on Information Technologies and Security, 2020, 12, 37–50Search in Google Scholar

Received: 2019-12-09
Accepted: 2020-08-04
Published Online: 2021-03-18

© 2021 Siraj Munir et al., published by De Gruyter

This work is licensed under the Creative Commons Attribution 4.0 International License.

Downloaded on 21.2.2024 from
Scroll to top button