decode sks (gpg) keyserver database format (Python)

Question

i'm doing research on the topic of the web of trust, therefore i need as many pgp keys as possible. I downloaded a key dump of a SKS keyserver and import everything to my local SKS keyserver. The sks keyserver uses berkeley DB as a database. Unfortunateley the SKS keyserver is written in OCaml and i cant figure out how to extract all keys from the database. I opened the database and read all keys, but i looks like the keys are in a weird (i read somewhere its radix64. i tried different radix64 decoder -> they all failed) format. Does anybody know a (pythonic) way to extract all keys and import them to mysql/neo4j/whatever?

Here is an example entry:

\x00\x99\x02\r\x04R\xb6\xcag\x01\x10\x00\xcc/\xd9\xd0\x11\xaa~b\xcc\xb0L\xee\x0ed!*\x8f\xa6\xc82\x98\xe6\x92\xf1\xb9\\*\xc2\xbb\xb2\x1b7m*V6\xe9\x8aOI\xcc\x9d\xf0\xb7\xe0\xa8\x90\xf6\x1aN6\xec0\x0f3\xf4=h\xe1_\xb2\xef=\xcb\xfe\xc6\xf2\x16\xac\xdb\xe3\x00\x05\xe8\x92S`Co\xc1\x88t\xaa\xabA\xcc\x1b\xd8\xeb\xf7\x86\x17F\xc7\xdd\xe25Y\xc4\x85\x92=p\x95C/\xf1>\xbe$Z\xbe\x81\xd1\xb4\xf3\xb4\x9d\x85\xc5\xc9\xbc\'s\x8bJw\x02\x1b[I\xcd\x1e\xc7\x1c\xc8\xebG\x8a\x9cx\xe18\x97l\xefC\xdaU\xc9\x86\xf5\x80!(\x89.\x1e\xe7\xa6csjDK\x08\xd5\xe2\x03\xd3.\xcc\xc2\xc1\xcb\x81@-\xf7r0=\x06\x17s\x13x`{\xea\x85w\xb0*\x9e\xf3{\x05\xbaj\xe3!\xf9f\xef\xc9\xb9v]\xf5\x8f\xf4\xd2\x99\xd4\xf6\x16\xc6\xc1n\xcd\xc1\xbc\xaa?\xbc\xbd\xcadd\x07\xdeF(\x8e}\x9f\x07b.\x0f\xa5\xab\xd7\x0c\xdbV\x93\xb4\x19ek\xd1\x10\x89\xa7\x98#\xad\xdf{\xa9\x8f\xc1bF\xa0\x11\xa8\xf7\xf6\x13La\xeb\xbb\xf8q\xa8\xae\x9b\xc5)\xdb2\xfa\x04\xc7\xb4\x8b\xfb \xd4k\xf0\x19X\'\xf6\xbd\xa4\xeb\x06\xb0\x91\xf2Z\xe9\xc9kD\xd7\x94#\x0f)\xfb\xa9\x90\xd3\x9a\x80b\x86`1\xfb\xa10\xae\xaa\x1e\x00.\xe8\xf3\xac]\x99w\nW\x97vC\x9f\x84\x85(\xe5\xc9\xef\x0e\x93\xfe\xe6\x07H9\x01\x17\xa9R3\x07\xef\xa7\xa2\xca\xbc\x9c\xddL\xe3\x15^}\x0f\xb6\t\xaa\x0fU\x97\rEZ&\xfb\xad\x13{%\xbf\x070O\x8f\xa4"\xb9\r\xfd\x05\xec/\xc0h\x9a\xbe\xe4w\x9b8\x86\xc2\x87\xfb\x1e\xee(mK\xdf=4EiC\xf4\xb0\x18\xbdZ\xaf\x8c;\xba\x8e\xb7\x19e\xe5*\xb6\xa5l\xa8\x95\xdd\x985\x81\xaaP\x0c~\x13\x1b\x90\x10\xc9\x1f\xd8Q\x01$\xb3\xe8t\x06\xbf\x81\x9e\x83k\xf8\xb7\x19\xfa\xd1u\x1c\x85\xf7\xa2\x87\x0ba\xc5\x1dz3K\\ \xd6\xabR[\xaf\xba\x92\xdc\x06 \x81EJ2\xbc\x89\x00\x11\x01\x00\x01\xb4*Laura Schmitz <laura.schmitz87@icloud.com>\x89\x02=\x04\x13\x01\n\x00\'\x05\x02R\xb6\xcag\x02\x1b\x03\x05\t\x07\x86\x1f\x80\x05\x0b\t\x08\x07\x03\x05\x15\n\t\x08\x0b\x05\x16\x02\x03\x01\x00\x02\x1e\x01\x02\x17\x80\x00\n\t\x10h\x1d:u;l$\x9e\xcb\x83\x0f\xfb\x06\xbe\x9c#7U\x12Y\x83k\x80@Z\xabT\xe8\xec\xfc;\x9f6\xeb\xad\xf7{\x9c\xcd\xd1\x81\x82\x81\xe9\x91\x0b\xdb3\xa8\x1cq\xe2r\x11\x11c3\xf59\xa4\x9e\tDp\xc7\xa7W;\x07#Q\xf2d<\xa6\xd3\xa9\xde\xfd\x86i\t\x93\xfePc\x019X\xb9\xb1\xf7(\x97\xf5<Cr\xf3\x15\x83\x98\xc2\xfb\xd2\x14\x84\xb0.\xefm,b\xe9C\x8d\x07\x81\xaa{\xcc2\xe5(,\xb1\xf2\xc1\xbe\t \xf1[==H\x0e\xdbi>A(\xa6\xdd!+U\x88\xc6\x1d\xe6\xdf\x8a\xa4\xa0\x00\x0f\x00@\xd6\xc40o\x82\\\x9e\x97\xa0\x85W\xf6g\x1e\x14\x1cA\x1d\x18\xf8\x96\x12\xf0\xeb\x83J4\xffB\xc4\x8eU\x00\xa8\x9d\xd4U\x9e\xc66\x11\xda\xe7\x06\xbaW\xc4\x16(\xaa\xf1\xe3=fR\x05K\x1e\x0f\xae\xd4n\xbd\x0c\\\xb8i\x11#U>\n\xaf \xce\x81G\xe0\xbf\x84\x1a\x81j\xfe\xec\xe08Y\x99\xfd\xd6\xc1MLZ\xb6k\xc2K\x03\xdbd\xba\x16T\xbaHJ\x80#\xc1(W\xdb\xfaKx\xf3\xdf\xe6\xb2M;\xa59\xe6g\x13\x8a\xdd\xd4\xef\x824A,\xf1\x10\x02\xcd\x10\x7f\x0b\xf3N\x10\x06\xef\xc7\xe2r\xa0\xf0\x93\xbf\xcbA|\xe9\xb5\xe5\x06\x1c\xcc\xed\xb2ii\x87\xc0y\x0b\n\xea\xac\xf9\xf6\x8a\xec\x1c~\xb4\x80\x0f\xc9\xb3{\xfd\x07\x0cwO~u\x10$u\xe1\xab.\x90\xd9d\xdf\xbc6\x89\xfdJ:\x01Jx\x1a\xec\xf0\xea\x14Z\xe1\xe83%\xaco\xa2\x91\xc7\xfd\xa9-c\xfc\xfb\x85\xd9ON\xd2F\x88\\\xe2\x84A/\xfd\x8c\x18\xc4\xad\xfb\x0b\xed\x7f\x9eKnn\xd4\xa1\xbc\x89\xa8\xa7\xc9\xa4\x81\x80&3\xdb\x8b\x04\xe0\xdf(\xa5\x90\xd6\xbe\xa2\x18\x01Y\xf4\x9f\xf0l:\xc5%\xf1\xe7\xa0\xd0\\\x16\n\x10\xe6\xf1s\xbfHL\x7f\x0b\x9cF\xcc9\xb5C\x0f\xce\xb0\xe1\t\xc7\x0f\xa1\xe81vr\xcd*\xc5^}v\xfd\x97\xb4^K\xaf\xdb\xe3\x0e:\xd7\x88\xcee\xc2\xd9\x08rP\xad|\xe5e\xc50x\x81\xed;g\xeb\x9c\xfc\xb9\x02\r\x04R\xb6\xcag\x01\x10\x00\xb1\xca\xa6\x84\xb0\x85\xe6S\xdf\x99d\x13\xd5\x92\xb7\xce\n~\x0f\xd1\xac\x1e`\xc5\xfd\x03\x0fA\x0b\x87G\x80f}au{\x18\xb3\x05\x9a\xd1\xf5\x98\xb8\xf5\xeb*\x0fpR4lw"\xc9\xd0L\x1ae \x87\xf2\x8d\x94\x9c\x98\x18k\xda\x92`\xab\xd6o\xeco\x91\xbd\xf3\xd2v\xa4\xe6\x96s\xe1\x83\xd3\xa4\xe81\x7fF\xea\x8a\xa8>-\xe4(\xdb\x92\x89\x8b\x15\xe5\x8e\xc5#M+\xa6\xb0\xae\xb6\xfa\xa2\xec-\x80\x9c\xb2nH\x9a\x81g\xec\x9an\x0c/}\xe9\x075\x93\xca?\xb3\xf8\x11-2\x0b\x7f\xcd\xb2\x84\xa6\t\xcd\xd1 \x14\xcb`\xb1\xa7\'\xdd\x88-\xab%\xb1|6\x85/\xce\xaeEFZ\xd0e\xf2\x0c\x870l\x87\xed\xf5d\xca\x90\xcd\x1d\xdd\nr\x11\x82\xe6\xa9w\nl\xd4\xda\xee\xa4l\x90\xe8\'\x01?\x18\x99\x10\x03\x95\xfd\x07\xbe\xb2bK\xa3\x07\x18\xf00\x16U\xdc\x84J\xcdd\xbd\xc1ZQ\x19\xbc\xdd+t\xf9$\xe5+\xdcg/\x95K\x87\x08\x81\x94\x16\x1b\x9f\x8f\xba\xfd=uR-\xfe\x01];vB\x97\x8e\x1e\xc6\xc5$\xa2n\x99k\xffdp\xc9:\x84\xb5\xb4\xd89\xc7\xb3\xcc\x8f\x18\x0c!\r\x83\xc6\x1a:j\xa4\xa4R\xa6\xd0\x82r\xe22o\xcazGw\xf1\x90\x85\xe5[\x96\xe6\x19\xc5\xf1\x87V\x9b\x9a\t\xb4\x82\r\xa1\xcc\xee\xe4\x1eV\xfc\xd2\xa6q\xa4W8\xb1^W\xf0\xe1\xdf9\xf2\x9c \xb0y\xb2\x94\xba\xdcD\xd0\xf7a\xf2\x00[\xc9\xe6-\xfdR\x91\x08\x17\x02-\x1e\x7f<\x85\xef\xef\xf2@\x0f\xa5\x93v\xa1{)\xef\xf2\xb4c\xbd\xc4@\x92\xd8\xf5Z\xe53\xd2\xad\xb7\x99\xf9\xbcX,\x08\xcd\xec9\x04Lr\xbdo|\x1f \x1d\n\xc1\n\xa9\x08E\x1dV\x18pn\x9c\xaa\x13n|o\xe2/\xd2G:\xba-\xa6\x88\xd1\x02\t\xbc\xcd\xfa\x04\xb8h\xe4\xd6tjz\x9d\xd0\xf0M\xa8\x93e\x18\xcf\x16"#L\xca\xe6\xb8!\x80\x93\x80\x8c\x0c\x855\x11\x99GbY\xf4\xe8\xb1\x95\x9b\xcd\x13\x00\x1fO\x9f\x00\x11\x01\x00\x01\x89\x02%\x04\x18\x01\n\x00\x0f\x05\x02R\xb6\xcag\x02\x1b\x0c\x05\t\x07\x86\x1f\x80\x00\n\t\x10h\x1d:u;l$\x9eKQ\x10\x00\xab\x8e*\xb73\xe4\xa6J\x88\xfd\x02\xed\xf4\x06Q\x17I5\xc8z\xdbe\xe37u\xebfjH\xde\x88\xa9\x13\x95k\xb2\x8cF\xc8M\x9b\xa5\x1b\xba\xd9\xeeMrk \xb0rG\xb1\x16\x92%\xd9[S\x05\xa5\x14\xc8\x0c*\x84\x95#\x0c/\xad>w\x00\xf2\x84\xb9\x80\xd0%\x9b~J\x9bH\x94\x87#\xc3O\xc7\xb3\xa3\xc3{\x01\xff\x03\x99A\x9a\xe8xZ\x96\xd6\xfa\x15\xe4T\xfd\xb3v\x87\xea\xb6\x80\xe9\xceA\xacTwp\xf6\xc7\x1f\xc3\xde\x855\xe3\xf3hZ\x1dN7\x00,\xc8\x0e y\x0c\x10\xd5\xde\xf8\xe9\xb5ld\xae\xcf\xfe\x0c\xea|\'HRc\xb1\x9a\x1c\xf1\xb0IM\x8b\xd3\xf3\xb0]\x86\xb8\xe8\xe2-\xc6%\x8d\xca\x0e4\x81\x1c\xd7\xf2\xb7\xd5\xa1h\x90`d.$\xcc\xb2\xb8\x7f\xf0\xa3W(\xa1\x82w\x97\x11P\xcb\xce9\x8c\x0en\xf3\xee\x85\xd1\xe8\xca!\x95\xf0h\tr\x11S\xaf\x04U\xd6\xe0\xc4\xc27\xb3\xa6_\xf1\x97\xa8bd.\x8e\x9c\n\xe8@%J\xc5\x9e\xf3\xf3\x8eng|\x9e\xcd\xbb$0C\xd4!\xd7\xc4\xff\x11\xc2.\xb2M)\xc8D\xb1\x03^3\xc7\x13\x8a:\xdbw\xbeZ\x86&#|Q9\xcfD2n\xc5`\xa8\xa4\xc4\x1a\xef\xe6\x16\xe8\xe1\x1f\xde\xabG\x17? v\'\xd0\xca\x0e\xe3\'~\xb3\xc9M\x7f\xd2[\xc4!5\xbb\x0c\xa1k\xa5\xf1V\xe03[\x92\xc8A0\xad\xe4\xc5\xb3\xe6\xdf\xd4\xc9\xef\xddjM\x0f\x06b-\x1d\x96\xdey7c\xbd\x8e\x12\xe3\x16O\xdf\x07\x98\xb6\xfe\xf1\x84\xdb9f\x8d\xbb\xd9\xe9:i\x15[\x0fb\x19m\xcd!s\xce\xb8\xbd\xacO\xbe\x9bN5qF\xa9w\xd7\xb0h\x8dXL#\x85\xbf\x10k\xd7\x87b\x88Q\xe8z#\xa3\x1d\x9d\xcc\xbe^\xae3\xf5\xfe,0\x99\xad{\xa0\x81\xc1\xd8\x05\x9b\xce\xff(E\xac\xc4\xed\x07%\xebH\x8e\xc2,\x16\xe6F\xbe\xc7\x8f\x1d\x97\n\x01\xa9\x06\xba\xd2\xf1=\x054:p\x10te\xd7\xb8}>\xb1\xdb\x077\xd7]\x0c\x05\xb5

Answer 1

I'm answering your original problem here, fetching the key set and working with it .

Does anybody know a (pythonic) way to extract all keys and import them to mysql/neo4j/whatever?

Consider using Hockeypuck instead of the SKS key server; a very recent implementation in the Go language, which can also sync with the wide spread SKS key server using the SKS protocol. But, as a big difference, the keys are stored in a relational database (namely PostgreSQL), making queries and ad-hoc analysis much easier.

No caring about transforming any data, just setup the key server and query it. You can even synchronize your working data set with the other key servers without hassles and expensive exports.