使用 Python 从 JavaScript 数组中删除重复项

Question

Suppose I have a JavaScript array of elements that looks something very similar to:

var oui = new Array({
    "pfx": "000000",
    "mask": 24,
    "desc": "00:00:00   Officially Xerox, but 0:0:0:0:0:0 is more common"
},{
    "pfx": "000001",
    "mask": 24,
    "desc": "Xerox  Xerox Corporation"
},{
    "pfx": "000002",
    "mask": 24,
    "desc": "Xerox  Xerox Corporation"
},{
    "pfx": "000003",
    "mask": 24,
    "desc": "Xerox  Xerox Corporation"
},{
    "pfx": "000004",
    "mask": 24,
    "desc": "Xerox  Xerox Corporation"
},{
    "pfx": "000004",
    "mask": 24,
    "desc": "Let's pretend this is a repeat"
   });

Imagine now that the file is very large, and some of the "pfx" values are repeated throughout the data set. Obviously manual de-duping is out of the question, so I'm trying to figure out the best way to approach it programmatically. How can I write a python script to read in the.JS file containing this data set to de-dupe and remove any duplicates? In other words, I would like to read in the JS file, parse the array, and produce another JavaScript file with a similar array, but only unique values for the pfx variable.

I've gone through a couple of other Stack Overflow questions that are similar in nature, but nothing seems to quite fit this case. In my python testing, I can rarely just get the pfx variables by themselves to remove the duplicates, or Python struggles to read it in as a proper JSON object (even without the "var" and "new Array" portion). I should also note, that the reason that I'm doing the de-duping in Python over another JavaScript function within the JS file (which I tried following examples like this ) is that it just inflates the size of the JavaScript that has to be loaded onto the page.

In the future, the array is likely to continue grow - thus to avoid unnecessary loading of JavaScript to keep page response times quick, I figured this was a step that could, and should, be performed offline, and added to the page.

For clarification, here is a model of the website I'm trying to mock up: https://www.wireshark.org/tools/oui-lookup.html . It is very simple in nature.

Research:

Convert Javascript array to python list?

Remove duplicate values from JS array

Answer 1

Since the structure is not nested, you can match the array with a regular expression, then parse it with JSON, remove duplicate objects with filter in Python, and then replace with the deduplicated JSON string.

Use array literal syntax ( [ and ] ) rather than new Array to keep things cleaner (best never to use new Array ):

import re
import json
str = '''
var oui = [{
    "pfx": "000000",
    "mask": 24,
    "desc": "00:00:00   Officially Xerox, but 0:0:0:0:0:0 is more common"
},{
    "pfx": "000001",
    "mask": 24,
    "desc": "Xerox  Xerox Corporation"
},{
    "pfx": "000002",
    "mask": 24,
    "desc": "Xerox  Xerox Corporation"
},{
    "pfx": "000003",
    "mask": 24,
    "desc": "Xerox  Xerox Corporation"
},{
    "pfx": "000004",
    "mask": 24,
    "desc": "Xerox  Xerox Corporation"
},{
    "pfx": "000004",
    "mask": 24,
    "desc": "Let's pretend this is a repeat"
   }];
'''

def dedupe(match):
   jsonStr = match.group()
   list = json.loads(jsonStr)
   seenPfxs = set()
   def notDupe(obj):
        thisPfx = obj['pfx']
        if thisPfx in seenPfxs:
            return False
        seenPfxs.add(thisPfx)
        return True
   return json.dumps([obj for obj in list if notDupe(obj)])

dedupedStr = re.sub(r'(?s)\[[^\]]+\](?=;)', dedupe, str)
print(dedupedStr)

Output:

var oui = [{"pfx": "000000", "mask": 24, "desc": "00:00:00   Officially Xerox, but 0:0:0:0:0:0 is more common"}, {"pfx": "000001", "mask": 24, "desc": "Xerox  Xerox Corporation"}, {"pfx": "000002", "mask": 24, "desc": "Xerox  Xerox Corporation"}, {"pfx": "000003", "mask": 24, "desc": "Xerox  Xerox Corporation"}, {"pfx": "000004", "mask": 24, "desc": "Xerox  Xerox Corporation"}];

If possible, you might consider storing the data in a separate tag, rather than being in inline Javascript - it'll be more maintainable. Eg, in your HTML, instead of

var oui = [{
    "pfx": "000000",
    "mask": 24,
    "desc": "00:00:00   Officially Xerox, but 0:0:0:0:0:0 is more common"
},{

consider something like

 var oui = JSON.parse(document.querySelector('[data-oui').textContent); console.log(oui);

 <script data-oui type="application/json">[{ "pfx": "000000", "mask": 24, "desc": "00:00:00 Officially Xerox, but 0:0:0:0:0:0 is more common" }]</script>

Then you don't have to dynamically change the Javascript, but only the <script data-oui type="application/json"> tag.