Documentation
Start typing to search…

Censys Universal Internet Dataset

The Censys Universal Internet Dataset provides accurate, up-to-date records about Internet-facing hosts, including information about the host as a whole and its services.

Host information falls into three categories:

  • Host-level data.
  • Service summary data.
  • Service data.

Host-level data

Host-level data includes information applicable to the host as a whole:

  • IPv4 or IPv6 address
  • name, if applicable
  • geolocation
  • routing information
  • operating system

A snapshot date also provides a timestamp from Censys for the representation of the host.

Host-level fields are located at the root level of the entity.

Example host-level data in JSON

{
    "host_identifier": {
      "ipv4": "61.239.213.42",
      "ipv6": null,
      "name": null,
    },
    "ipv4_int": "1039127850",
    "ipv6_int": null,
    "snapshot_date": "2021-05-23 00:00:00 UTC",
    "location": {
      "continent": "Asia",
      "country": "Hong Kong",
      "country_code": "HK",
      "city": "Shatin",
      "postal_code": "",
      "timezone": "Asia/Hong_Kong",
      "province": "Sha Tin",
      "coordinates": {
        "latitude": "22.3829",
        "longitude": "114.1779"
      },
      "registered_country": "Hong Kong",
      "registered_country_code": "HK"
    },
    "autonomous_system": {
      "asn": "9269",
      "description": "HKBN-AS-AP Hong Kong Broadband Network Ltd.",
      "bgp_prefix": "61.239.192.0/18",
      "name": "HKBN-AS-AP Hong Kong Broadband Network Ltd.",
      "country_code": "HK",
      "organization": null
    },
    "operating_system": null
}

Service summary data

Summary data includes fields that inventory the services Censys global scanning engine detected on the host, presented as two lists: one of the open ports and one of the service names.

Each index in the ports_list array is paired to the value with the same index in the services_names_list array.

Summary fields are located at the root level of the host entity.

Example service summary in JSON

{ 
  "ports_list": [ 
  "7547"
  ], 
    "service_names_list": [ 
      "HTTP" 
    ], 
}

Service data

Services is the bulk of host data, with a separate record to show each service scanned by Censys global scanning engine.

Identifying information and metadata

Within a service record, a few fields can act as a composite identifier:

  • Port number
  • Transport protocol (for example, TCP, UDP, QUIC)
  • Service name (or extended service name, if you prefer to see the TLS indicator on the name

Service names usually reflect application-layer (OSI-model L7) protocols such as HTTP or AMQP, although they can also be app-specific services that run on top of HTTP, such as Kubernetes or CWMP.

Example service-identifying data in JSON

"services": [
  {
    "port": "7547", 
    "transport": "TCP", 
    "service_name": 
    "HTTP", 
    "extended_service_name": "HTTPS", 
    ... 
  } 
    ]

Metadata

Metadata includes information about when and from where Censys global scanning engine observed the service:

  • Perspective: the name of the ISP that Censys global scanning engine peered with
  • Source IP
  • Observation timestamp
  • Discovery method: the name of the method by which a service was first found

Example metadata in JSON

"services": [ 
  { ... "perspective": "TELIA", 
   "observed_at": "2021-05-23 16:11:27 UTC", 
   "source_ip": "74.120.14.38", 
   "discovery_method": "IPV4_WALK_FULL_PRIORITY_1" 
   ... 
  } 
]

Service-specific scan data

Service-specific data includes a map of parsed scan data that corresponds to the service name, TLS data, a banner, if applicable, and software information.

Service name

The value of the service_name is a key for a record containing all of the service-specific data parsed from scan.

For example, an RDP service can have a service_name of RDP and a populated record whose key was rdp containing data obtained by the Censys RDP scanner.

TLS

Transport Layer Security (TLS) is shown as a separate service-level record because it can be enabled for many service types.

Software

Software is a repeated record that contains application, operating system, and/or hardware information presented according to the CPE version 2.3 standard.

Example service data

"services": [ 
  { "tls": null, 
   "banner": "SFRUUC8xLjEgNDA0IE5vdCBGb3VuZApDb250ZW50LVR5cGU6IHRleHQvaHRtbApTZXJ2ZXI6IFdlYlNlcnZlci8xLjAgVVBuUC8xLjAKRXh0OiA=", 
   "http": { 
     "request": { 
       "method": "GET", 
       "uri": "http://61.239.213.42:7547/", 
       "headers": [ 
         { "key": "Accept", 
          "value": { "headers": [ 
            "Ki8q" 
          ] 
                   } 
         }, 
         { 
           "key": "User-Agent", 
           "value": { 
             "headers": [ 
               "TW96aWxsYS81LjAgKGNvbXBhdGlibGU7IENlbnN5c0luc3BlY3QvMS4xOyAraHR0cHM6Ly9hYm91dC5jZW5zeXMuaW8vKQ==" 
             ] 
           
           } 
         } 
       ], 
       "body": null 
     }, "response": { 
       "protocol": "HTTP/1.1", 
       "status_code": "404", 
       "status_reason": "Not Found", 
       "headers": [ 
         { 
           "key": "Content-Type", 
           "value": { 
             "headers": [ 
               "dGV4dC9odG1s" 
             ] 
           } 
         }, 
         { "key": "Server", 
          "value": { 
            "headers": [ 
              "V2ViU2VydmVyLzEuMCBVUG5QLzEuMA==" 
            ] 
          } 
         }, 
         { 
           "key": "Ext", 
           "value": { 
             "headers": [ 
               "" 
             ] 
           }
          ]         
         } 
       ], 
       "html_tags": [
        "PHRpdGxlPk9iamVjdCBOb3QgRm91bmQ8L3RpdGxlPg==" 
       ], 
       "body_size": "189", 
       "body": "PGh0bWw+CjxoZWFkPgo8dGl0bGU+T2JqZWN0IE5vdCBGb3VuZDwvdGl0bGU+PC9oZWFkPjxib2R5Pgo8aDE+T2JqZWN0IE5vdCBGb3VuZDwvaDE+VGhlIHJlcXVlc3RlZCBVUkwgJy8nIHdhcyBub3QgZm91bmQgb24gdGhlIFNlcnZlci48cD5SZXR1cm4gdG8gPEEgSFJFRj0iIj5sYXN0IHBhZ2U8L0E+PHA+CjwvYm9keT48L2h0bWw+" }, "body_hash": "c2hhMTozZjdhODk0YWI0NDhkNWFmNTlmN2M2ODQxY2Y2YjZkMGZkOTg2M2Rm" 
   }, 
   "software": [ 
     { 
       "uniform_resource_identifier": "cpe:2.3:a:hikvision:hikvision_web_server:*:*:*:*:*:*:*:*", 
       "part": "a", 
       "vendor": "Hikvision", 
       "product": "Hikvision Web Server", 
       "version": null, 
       "update": null, 
       "sw_edition": null, 
       "target_sw": null, 
       "target_hw": null, 
       "language": null, 
       "component_uniform_resource_identifiers": [], 
       "other": [], 
       "edition": null, 
       "source": "OSI_APPLICATION_LAYER" 
     } 
   ], 
   "vnc": null, 
   "rdp": null, 
   "ssh": null, 
   "mysql": null, 
   "ipmi": null, 
   "amqp": null, 
   "elasticsearch": null, 
   "kubernetes": null, 
   "memcached": null, 
   "mssql": null, 
   "oracle": null, 
   "prometheus": null, 
   "redis": null, 
   "snmp": null, 
   "postgres": null, 
   "mongodb": null, 
   "bacnet": null, 
   "dnp3": null, 
   "dns": null, 
   "ftp": null, 
   "imap": null, 
   "ipp": null, 
   "modbus": null, 
   "mqtt": null, 
   "ntp": null, 
   "pc_anywhere": null, 
   "pop3": null, 
   "s7": null, 
   "smb": null, 
   "smtp": null, 
   "telnet": null, 
   "fox": null, 
   "openvpn": null, 
   "coap": null, 
   "truncated": false, 
   "sip": null, 
   "team_viewer": null, 
   "x11": null, 
   "skinny": null, 
   "unknown": null, 
   "cwmp": null 
  } 
] 
}

Updated 6 months ago