Skip to content

Import data hub

Jump to bottom Edit New page
ZLightning edited this page Nov 9, 2014 · 25 revisions

Streaming Live Views into mal4s Send continuously updated data to a lobby or common area Mal4s monitor display.

####Opening mal4s files automatically or by downloading

Ideally when you are looking at some Internet domain name or netblock data on a web page you can just click on the data and instantly have it displayed in the mal4s. (You can do this right now with Mac and Ubuntu with Firefox or Chrome.)

Open .mal4s files automatically: Mac

Open .mal4s files automatically: Ubuntu

####Formatting data for mal4s

If you have a pre-formatted data file, send it into Mal4s using STDIN, such as:

cat myfile.txt|mal4s

You can also open the file like this now too:

mal4s myfile.txt

If you are not working with data that someone else has pre-formatted for Mal4s, what follows are the details of the input file format expected.

You can add extra fields and customize the display of the fields in the hover text box with custom labels. Mal4s isn't limited to displaying data about Internet hosts and malware - feel free to try it with any type of hierarchal data.

####Do you have data you'd like to have magically formatted for Mal4s?

Here are a few options for painless formatting:

  • Our service that converts a pasted list of domains into a *.mal4s file

  • Our service that accepts pcaps (packet capture files) and automatically extracts the network activity, formatting it for the httmp.conf and dnsm.conf. Files named with an MD5 hash will be looked up for malware type.

  • Post a sample of your data as a github gist and then share the link to your gist with us in an issue for community help.

  • Subscribe to a custom Amazon ec2 instance with Mal4s data processing built in

  • Hire us to write a custom parser and data gathering script to turn your live data into a Mal4s display

Send email to data at dissectcyber dot com to request access to automated parsing of your data.

####Format of *.mal4s files:

Description of mal4s file fields

Mal4s files are plain text and contain one record per line. Each line causes the display of one colored ball in the screen which carries with it all the details to display in the text box that appears when hovering your mouse cursor over that ball.

Mal4s lines have three types of fields:

  • required by the program but cannot be displayed

  • causes a branching of the cluster tree and can be displayed

  • non-branching, used for displaying info only

Here are some examples of the formats we use currently for various types of data. The first is for new authoritative name server host lists, next is http traffic of malware samples, and the last is dns traffic of malware samples. While Mal4s is handy for Internet hosts, we believe it can also be used with non-Internet data such as finding clustering in any type of textual data.

####newns 1388892986|BIGROCK SOLUTIONS LIMITED|A|arin/US/198/56/238/61/NS1.COOLRINGSONLINE.COM~NS2.COOLRINGSONLINE.COM/Enzu Inc/NS2.COOLRINGSONLINE.COM|C638EE|us|18978|subpoena.coolringsonline.com|BIGROCK SOLUTIONS LIMITED|2006-07-18|OK|United States

####httpm 1386850224|Troj~Bckdr-RAK|A|ripencc/PT/195/22/26/231/ns1.csof.net/ClaraNET LTD/uupqgmlru.net|C3161A|pt|8426|anubisnetworks.com|GODADDY.COM LLC|2013-12-03|80 GET|200 OK|[SLASH]||39cce9206bf0724188fee82efb7d4388

####dnsm 1386849853|DomainIQ pay-per install|A|arin/US/4/2/2/3/ns5.tguhost.com/Level 3 Communications/staticrr.paleokits.net|C8C8C8|us|3356|c.resolvers.level3.net|SOLUCIONES CORPORATIVAS IPSLU|2011-06-03|53|A|NOERROR|staticrr.tgusrv.com,85.12.8.28|3b1369b3b02b81e0609444d7a2c4e5fa

####NEEDS UPDATING BELOW HERE TO MATCH NEW FEATURES:

EPOCH|REGISTRAR|A|RIR/COUNTRY/1/2/3/4/REGISTRATION DATE/REGISTRAR/example.com|HEXCOLOR|USERIMAGE|COUNTRYNNASNNN|NONBRANCHING2

The last two are optional.

#####Example:

1369141249|GoDaddy.com LLC R171-LRMS|A|apnic/JP/106/187/94/116/2012-04-11/GoDaddy.com LLC R171-LRMS/vyqhdtnsfrie.info|6ABB5E|jp|JP2516

#####Detailed explanation of fields:

######EPOCH

A timestamp representing the number of seconds since the year 1970. Here the value controls the speed and chronology of how the display of your clusters progresses. Don't use timestamps that are years apart unless you are skipping dead space in the configuration variables. At present we just increment the datalines so the EPOCH is unique on which one and controls the sort order.

######REGISTRAR ${plotter}

The domain name registrar company

######A

This field may be used to Add, Delete, or Modify the dot on the screen for this host. For simplicity for now you can set it to A for Add.

#####Branching Fields:

Each forward slash in the data creates another offshoot branch on the tree. This is what controls how hosts on the same branch sequence cluster together.

######RIR ${b1}

The first branching field is like the first level categorization. Notice that Mal4s plots from an original center point; this is the point from which each of the first category levels branches off.

In our data we use RIR for the netblock where the IP address of the host is pointed as the first branching field. RIR is the Regional Internet Registry such as arin, ripe, apnic, afrinic, lacnic etc.

######COUNTRY ${b2}

We use the two letter country code for the netblock where the IP address of the host is pointed as the second branching field.

######1/2/3/4 ${b3}/${b4}/${b5}/${b6}

The octets of an IPv4 address separated by forward slashes instead of dots, or try your preferred IPv6 format here.

######REGISTRATION DATE ${b7}

The registration date of the Internet domain name.

######REGISTRAR

The registrar name for the Internet domain name is repeated here. Without the placement here in the forward slash separated path data, there would be no clustering / gathering / branching by registrar.

######ADDITIONAL BRANCHING FIELDS

You can add more branching fields such as authoritative name server host, company name from IP whois etc. Separate each piece of data you want to cause an additional branch with a forward slash /.

In theory there's no limit on the number of branching fields where that number would also be practical. However in practice we feel sure the limits of your computer would be hit fairly rapidly if you tried a crazy amount of branching fields.

######example.com ${host}

This is the end of the branching fields and is usually an FQDN that is the subject of the record. Note that an additional addressable field is automatically created called ${tld} based on any text to the right of the last dot in this field

####Non-Branching Fields:

######HEXCOLOR

The HEXCOLOR is like any other RGB hex color, just like the colors used in HTML. Don't put a hash # in front of the 6 digits.

Here's how we produce the colors in bash shell scripts for IPv4 addresses:

COLOR1=`echo "obase=16; ${OCTET1}"|bc`
COLOR2=`echo "obase=16; ${OCTET2}"|bc`
COLOR3=`echo "obase=16; ${OCTET3}"|bc`

The color field value itself can't be addressed or displayed.

######USERIMAGENAME

The user-image-dir is specified in the main config file. For example we have a dir name flags and in it is a file named for each two letter country code. They are all png files named such as fr.png for France's flag. The first field after the color will be ignored if your main config file has no user-image-dir defined, or will show the image if you have properly named images and the user-image-dir defined. 

This field cannot be addressed or displayed.

######ASN ${n1}

Here we usually use the Autonomous System Number for the IP address. Although country code and ASN do not have a direct relationship to each other, people do seem to find it useful to get a handle on both pieces of info with a count.

Mal4s is now capable of parsing an unlimited number of non-branching fields after the plotter image name. Each field must be separated by a | pipe symbol.

See formatting hover text to display the data you wish to see when hovering your mouse cursor over each ball.

####Obtaining RIR, Country Code and ASN

On Ubuntu:

apt-get install cpanminus

cpanm Net::Abuse::Utils

cpanm AutoLoader

(This doc is in progress, need to add the simplified script here to gather info from Net::Abuse::Utils for mal4s)

####Captions on timeline

Format caption files with an epoch timestamp as the first field and the desired caption as the second. The delimiter must be a pipe symbol |. The epoch timestamp determines when the caption will appear. Captions will build up on the screen unless the caption time to display expires prior to the appearance of the next caption.

A simple way to try out captions is to copy the first 2 fields of a *.mal4s file to a *.captions file. Here we save only the first appearance of each item to the captions file:

cut -d"|" -f1,2 sample--newns.mal4s|sort -u -k2 --field-separator="|"|sort -n > sample--newns.captions

mal4s --caption-file sample--newns.captions

Caption configuration settings are detailed in the Mal4s configuration file wiki page here.

hallstatt

Add a custom sidebar

Clone this wiki locally