Dataset Differences: Censys Platform vs. Legacy Search

The Legacy Search dataset includes hosts and virtual hosts. In Legacy Search, hosts are identified by an IP address. Virtual hosts are identified by a name and an IP address.

Virtual hosts are not present in the Platform dataset. Instead, internet assets that respond to hostname-based scans are classified as web properties.

In the Platform, web properties are identified by a hostname and a port. Hostnames can be name-based records (such as app.censys.io) or IP-based records (such as 104.18.10.85). Example names of web property records include the following:

• app.censys.io: 443
• 104.18.10.85: 8880

Web properties provide a more accurate, up-to-date, and comprehensive view of name-based assets than virtual hosts. Classifying and presenting web assets as web properties enables Censys data to:

• Better identify and showcase name-based assets, linking domain names and related services directly to their underlying infrastructure.
• Support data from multiple HTTP endpoints.
• Make searching for name-based assets feel like using a web browser.
• Dramatically improve the freshness of web data. Web properties are refreshed daily.
• Provide deeper visibility into application scans and the true footprint of internet assets, enhancing precision in asset discovery and threat monitoring.

How web properties differ from host services

Web properties offer insight into HTTP services beyond layer 7 while abstracting away HTTP protocol semantics.

• Web properties contain all records that correlate to HTTP-based scans.
• A web property can have one or more endpoints, which serve as distinct entry points to different resources or functionalities within the web property.
  • Each endpoint may be associated with specific applications, services, or APIs and can be individually monitored and analyzed.
• Web properties support deep scan information for HTTP-based scanners.

When to search across web properties instead of host services

Search web properties when:

• You want results that include hostnames.
• You are targeting software that runs on top of HTTP, such as WordPress, pprof, Kubernetes, elastic search, and so on.
• You are targeting software that services HTTP, like Apache or nginx.
• You need HTTP body information.
• You need data from endpoints other than /.

Do not use web properties when:

• You want results that include IP addresses.
• You are searching for DNS data, whois data, geolocation data, or routing data.
• You are searching for hosts serving HTTP as well as non-HTTP protocols.

Previous limitations with virtual hosts in Legacy Search data model

Censys has historically scanned hosts with HTTP services running on them. There were several limitations with this approach:

• In the Legacy Search data model, scanning hosts with HTTP services typically targets a single HTTP endpoint per host. By default, Censys aims to scan the root path (/) on HTTP and HTTPS ports (e.g., 80, 443) for each host. This approach makes it challenging to extract information from multiple endpoints like /wp-admin or /login.
  • Additionally, endpoints beyond the root may contain dynamic or context-specific content that varies based on parameters like session states, cookies, or user-agent headers. Scanning multiple or application-specific paths requires explicit scan configuration, reducing the ability to explore deeper or secondary paths systematically, which could result in missing critical information hosted on these endpoints.
• If a host had an HTTP service running on 443 but Censys could also identify a Cobalt Strike application running on that HTTP service, the Legacy Search data model couldn’t support showing both services, which run on 443. This led to difficult decisions about prioritizing scan results which didn’t feel aligned with our mission.
• Some users struggle to understand when and whether to include or exclude virtual hosts as targets of their search queries.
• Data on virtual hosts could be up to 45 days stale, making them less useful.

In the Platform dataset, some field names have changed, moved, or been removed. The following are examples of popular fields that moved betweenLegacy Search and the Platform. A complete list of fields that have moved or changed is available here.

The data model featured in the Censys Platform surfaces new, actionable context on what exactly mapped services and devices are, how they are configured, whether they are vulnerable, whether they are malicious, and where they live.

This new context allows users to:

• Quickly find a specific type of host, web property, or service.
• Take advantage of uniform metadata fields like CPEs, software, hardware, and labels.
• Quickly and easily conduct investigations and incorporate Censys data into automation workflows.

The following new context types are available on hosts and web property records. Not all context will be available to all users.

• Hardware
• Software
• Operating System
• Threats
• Vulnerabilities
• Exposures
• Labels
• Misconfigurations

Hardware

By extracting hardware into its own object, the Platform dataset is able to provide details about the hardware itself (such as a Juniper Router) while also listing any known information about the hardware’s components, such as the processor type and firmware. This provides a structure to give more context about the kinds of devices that may be vulnerable.

Software

By extracting software into its own object, Censys data captures specific details about the software itself (such as Microsoft SQL Server) while also including any relevant information about its components, like version numbers on hosts and web properties.

This structured approach allows the Platform to offer more context on software configurations and versions that may present vulnerabilities or security concerns, as well as highlight associations with other software or hardware dependencies, helping users understand the potential exposure or compatibility across diverse systems.

Threats

In the Platform data model, Censys defines “threat” using NIST’s primary definition of “cyber threat”:

Any circumstance or event with the potential to adversely impact organizational operations (including mission, functions, image, or reputation), organizational assets, or individuals through an information system via unauthorized access, destruction, disclosure, modification of information, and/or denial of service. Also, the potential for a threat-source to successfully exploit a particular information system vulnerability.

**Threats are limited in scope as they are new objects and will be made available starting in 2025. **The fields in the threat object provide a comprehensive view of threat information detected on services on hosts and web properties. Each field contributes to the detailed profiling and validation of potential threats, leveraging both Censys proprietary methods and industry-standard practices.

Changes to labels between Legacy Search and the Platform

In the Legacy Search dataset, labels are used for multiple purposes, ranging from indicating software manufacturers to describing records using descriptors like “network.device” or “login-page.”

There are fewer label values in Platform than in Legacy Search. This is partially because “labels” in Search 2.0 that were actually unstructured software, hardware, or operating system data have been moved to the appropriate component fields (e.g., jQuery, bootstrap) in the Platform.

The table below lists a few of the labels available in Platform:

IP host data model example
The following is an example of a host record in the Platform data model. A host represents strictly host-level information about an IP address. This includes its location, routing, and IP whois enrichments. All context included about a host is derived from service scan data.

{
    "dns": {
        // DNS Names and Forward DNS Data
    },

    "ip": <ipv4 or ipv6 address>,
    "service_count": <count of services running on host>,
    "truncated": <true or false>,

    "location": {
        // Location details such as continent, country, country_code, city, postal_code, timezone, and co-ordinates
    },

    "routing": {
        // Routing information such as ASN,BGP Prefix, BGP name and country
    },

     "services": [
        {
         "port": <port #>,
         "protocol": <protocol name>,
         "transport_protocol": <transport protocol - TCP / UDP, QUIC>,
         "misconfigs": [],
         "exposures": [],
         "vulns": [],
         "software": [],
         "hardware": [],
         "operating_systems": [],
         "threats": [],
         "labels": [],
         "ip": <ipv4 or ipv6 address>,
         "scan_time": "2024-10-20T20:07:50.000Z",
         "banner": "",
         "banner_hash_sha256": "",
         "<service specific details>": {
            // Details about service repeated for each identified service
        }
     }
   ],
    "whois": {
        // WhoIS data about host includes name, CIDRs, organization, contacts
},

    "labels": []
}

Search 2.0 host data model example

The Search 2.0 data model provides cohesive records about individual IPv4 hosts, decoupled ports, and protocol data. The 2021 model presents top-level information about hosts (IP address, location, routing information) and then an array of statically defined services.

In the Search 2.0 data model, Censys combined all information about a specific protocol in a single record.

For example, in the Search 2.0, the host “8.8.8.8” is presented as follows:

{
          “ip”: ”8.8.8.8”,
          “services”: [
                    {
                        “port”:80,
                        “service”:”http”,
                        “http”:{
                                  “title”: “Hello World!”
                     }
          ],
          “location”:{
                        “city”: “Ann Arbor”,
                      ...
          },
          ...
}

If, for example, this had been a MySQL service, there would be a “mysql” subrecord instead of an “http” one.

IIP web property data model example
The following is an example of a web property presented in the IIP data model.

{

    "hostname": <host name>,
    "port": <port number>,

     "endpoints": [ // repeat section for each endpoint on the asset 
        {
            "hostname": <host name>,
            "port": <port number>,
            "path": <path name>,
            "endpoint_type": "HTTP",
            "transport_protocol": "TCP",
            "scan_time": "2024-10-12T12:22:18.167Z",
            "banner": <banner details>,
            "banner_hash_sha256": <banner sha256 hash>,
            "http": {
                "supports_http2": false,
                "uri": <end point URI>,
                "protocol": "HTTP/1.1",
                "status_code": <status code of endpoint>,
                "status_reason": <status reason>,
                "headers": {
                          // Location and location headers 
                          // Content length and headers 
                          // Server headers 
                          // Date headers
                          // Cache-control headers                 
                 },
                "html_tags": [
                          // List of HTML tags
                 ],
                "body_size": <size of body>,
                "body": <body information>,
                "favicons": [
                         // List of favicons 
                         // Includes size, name, hash_sha256, hash_md5
                ],
                "body_hash_sha256": <body sha256 hash>,
                "body_hash_sha1": <body sha1 hash>
            }
        },
    "exposures": [],
    "hardware": [],
    "labels": [],
    "misconfigs": [],
    "operating_systems": [],
    "scan_time": "2024-10-12T12:23:13.926Z",
    "software": [],
    "threats": [],
    "vulns": []
}