Data acquisition is the foundational step in any Enterprise Perception System (EPS), involving the gathering of raw data from various sources—both machine-driven and human-driven. This data forms the foundation for perceptual analysis, decision-making, and the generation of insights. Historically, organizations primarily relied on human-generated data for decision-making, but with the advent of advanced sensors and automation technologies, machine-generated data has increasingly become the norm. The diversity of data sources, from automated sensors to human inputs, makes EPS flexible and adaptable in different contexts. This section will cover the types of data acquisition, the evolving role of both machines and humans as sensors, and the technologies that support this process.
In an Enterprise Perception System, data comes in a variety of formats. These can be broadly categorized into structured, semi-structured, and unstructured data. Understanding the data formats collected by an EPS helps to design systems that effectively integrate, process, and analyze this information as each format requires the use of different technologies.
Structured data refers to data that is highly organized and easily searchable in databases or tables. It typically conforms to a specific format and schema, making it easier to store, retrieve, and analyze.
EXAMPLES:
• Customer Records: Detailed information about customers, such as names, addresses, contact details, and purchase history, stored in relational databases like CRM systems.
• Bank Transactions: Transactional data such as deposit and withdrawal amounts, timestamps, and account numbers, stored in highly structured formats in financial databases.
• GPS coordinates: Location data collected from devices such as smartphones or vehicles.
ADVANTAGES:
• Easy to store and process in relational databases or structured query systems.
• Ideal for automated data processing and integration with other systems.
Semi-structured data does not follow a rigid structure but still contains organizational elements like tags or markers that make it easier to process compared to unstructured data.
EXAMPLES:
• Crawling websites: Data collected from web scraping or crawling often comes in semi-structured formats like HTML, where some sections (e.g., tags, metadata) are structured, but the content itself can vary widely.
• JSON or XML data: Common in APIs, where responses from sensors or software systems might contain information in structured fields but with variable formatting.
• Time-series data: Sensor readings over time, which often include a mix of structured (e.g., timestamps) and semi-structured components (e.g., varying attributes collected at different times).
• System event logs: Often semi-structured, containing identifiable sections but varying in content (e.g., different event types may log different data fields).
ADVANTAGES:
• Offers flexibility in data storage while retaining enough structure to allow for meaningful analysis.
• Useful for aggregating data from a variety of sources with different formats.
Unstructured data consists of information that does not follow any specific format or structure, making it more difficult to store and analyze. However, it often provides rich, context-heavy insights.
EXAMPLES:
• Images and video feeds: Captured by security cameras or other monitoring devices, used in computer vision applications.
• Text data: Transcripts of voice descriptions, user feedback, emails, or social media posts. Often analyzed with Natural Language Processing (NLP) techniques to extract insights.
• Audio data: Voice recordings from customer support calls, human descriptions of events, or even sensor-generated sound data.
ADVANTAGES:
• Rich in context and can provide deeper insights that are not possible through structured data alone.
• Essential for complex, perceptual tasks such as facial recognition, sentiment analysis, or audio pattern recognition.
Storing and Indexing Data for retrieval
Data Quality and Accuracy
Data acquisition in an EPS can be broadly categorized based on the nature of the data and the type of “sensor” collecting it. Both machines and humans contribute valuable data, though their methods of collection and the nature of their inputs differ. The balance between human-generated and machine-generated data in organizations has shifted dramatically in recent decades.
The Traditional Approach Historically, most enterprises and industries relied heavily on human-generated data to drive their decision-making processes. This was due in part to the lack of automated systems that could provide real-time data at scale. As a result, organizations were structured around human inputs, where observations, reports, and assessments were the primary sources of insight.
Examples include:
Call notes and descriptions of events: In industries like security, retail, or logistics, human observers were often the only means of recording events or tracking interactions. A security officer would log an event in a report, or a sales associate would describe a customer’s experience.
Manual reports or annotations: Workers in manufacturing or operations logged observations, incidents, and metrics manually. These reports captured qualitative nuances—like product defects or safety issues—that could be critical for decision-making.Surveys and forms: Feedback was often collected through structured forms or surveys, whether for employee performance evaluations, customer satisfaction, or compliance auditing. This method, while useful, often introduced delays and biases in the reporting process.
With the advent of modern sensing technologies, organizations began shifting from reliance on human-generated data to machine-generated data. The rise of digital sensors, IoT devices, and automation systems enabled enterprises to collect data in real time, at a scale and speed that was previously impossible. Machine-generated data provided an objective, consistent, and often more reliable source of information, enabling more accurate and timely decision-making.
Examples include:
Bluetooth tracking:
Systems that detect the presence of a Bluetooth-enabled device when it enters a specific area. This has been widely adopted in retail, security, and even smart offices to automate location-based tracking without the need for manual logs.
Temperature readings:
Sensors in factories, warehouses, or even office buildings provide continuous environmental data, ensuring conditions like temperature, humidity, or air quality are within desired ranges.
Motion detection:
Cameras and motion sensors automatically detect and log physical movement in spaces like warehouses, hospitals, or smart cities, reducing the need for human surveillance and manual entry of data.
