从任意和深度嵌套的 JSON （带数组）中过滤属性的通用方法

Question

Goal

I want to develop a middleware in TypeScript that filters the response of a REST API and returns only defined properties. It should work generically, ie independent of specific entities. Neither their properties nor the exact depth (eg with any number of relations) should be necessarily known.

Example

An author has any number of articles with any number of comments.

[
    {
        "name": "John Doe",
        "email": "john@doe.com",
        "articles": [
            {
                "title": "Lalilu 1",
                "text:": "la li lu",
                "comments": [
                    {
                        "author": "Bendthatdict Cumberstone",
                        "text": "Great article!"
                    },
                    {
                        "author": "Bendthatdict Cumberstone",
                        "text": "Great article!"
                    }
                ]
            },
            {
                "title": "Lalilu 1",
                "text:": "la li lu",
                "comments": [
                    {
                        "author": "Bendthatdict Cumberstone",
                        "text": "Great article!"
                    },
                    {
                        "author": "Bendthatdict Cumberstone",
                        "text": "Great article!"
                    }
                ]
            }
        ]
    },
    {
        "name": "Jane Doe",
        "email": "jane@doe.com",
        "articles": [
            {
                "title": "Lalilu 1",
                "text:": "la li lu",
                "comments": [
                    {
                        "author": "Bendthatdict Cumberstone",
                        "text": "Great article!"
                    },
                    {
                        "author": "Bendthatdict Cumberstone",
                        "text": "Great article!"
                    }
                ]
            },
            {
                "title": "Lalilu 1",
                "text:": "la li lu",
                "comments": [
                    {
                        "author": "Bendthatdict Cumberstone",
                        "text": "Great article!"
                    },
                    {
                        "author": "Bendthatdict Cumberstone",
                        "text": "Great article!"
                    }
                ]
            }
        ]
    }
]

Now I want to specify that it should return everything except the "text" of each article and the "author" of each comment.

Syntax could look like this with glob notation:

select("*,!articles.text,!articles.comments.author")

Approach

For objects and nested objects it is quite simple, eg with pick() and omit() of "lodash", but I fail when arrays step into the game. I did some research and came across packages such as json-mask , node-glob or glob-object but none of them exactly met my needs and I was not able to combine them for success.

Question

What is the most efficient way to generically filter an arbitrarily nested JSON with any number of further objects / arrays? Also, how could the TypeScripts type system be used to advantage?

I would be very grateful for general coding approaches or even tips for a package that can already do this!

Answer 1

In short I would break this up into functions. You could create helpers that do more or less what you want with a string/filter as you show however I'd work it in reverse. Get a nice way to iterate so any post processing can be done, then build your helpers as you wish against that. Here's what I mean:

Example

export interface IComment {
  author: string;
  text: string;
}

export interface IArticle {
  title: string;
  text: string;
  comments: IComment[];
}

export interface IComposer {
  name: string,
  email: string,
  articles: IArticle[];
}

// Remove items from list for brevity sake...
const authorList = [
  {
    "name": "John Doe",
    "email": "john@doe.com",
    "articles": [
      {
        "title": "Lalilu 1",
        "text": "la li lu",
        "comments": [
          {
            "author": "Bendthatdict Cumberstone",
            "text": "Great article!"
          }
        ]
      }
    ]
  }
] as IComposer[];


/**
 * Accepts JSON string or array of type.
 *
 * @param arr a JSON string containing array of type or array of type.
 */
export function selectFrom<T extends Record<string, any>>(arr: string | T[]) {

  // If you want to use this route I would suggest
  // also a function to validate that the JSON is
  // shaped correctly.
  if (typeof arr === 'string')
    arr = JSON.parse(arr);

  const collection = arr as T[];

  const api = {
    filters: [],
    register,
    run
  };

  /**
   * Register a search op.
   * @param fn function returning whether or not to filter the result.
   */
  function register(fn: (obj: T) => Partial<T>) {
    if (typeof fn === 'function')
      api.filters.push(fn);
    return api;
  }

  /**
   * Run registered ops and filter results.
   */
  function run() {

    return collection.reduce((results, obj) => {

      let result = obj;

      // Don't use reducer here as you can't break
      // and would unnecessarily loop through filters
      // that have no need to run, use for of instead.
      for (const filter of api.filters) {

        // if we set the result to null 
        // don't continue to run filters.
        if (!result) break;

        // Pipe in the previous result, we start with
        // original object but it's shape could change
        // so we keep iterating with the previous result.
        const filtered = filter(result);

        // update the result.
        if (filtered)
          result = filtered;

      }


      if (result)
        results.push(result);

      return results;

      // If changing the object you're going to 
      // end up with partials of the original
      // shape or interface.

    }, [] as Partial<T>[]);

  }

  return api;

}

Usage

By making this function based at the core you have a lot more flexibility. From there you could make a simple helper that maps your Glob or SQL like string to the pre-defined filter functions. Let me know if you have further questions.

const filtered = 
  selectFrom(authorList)
    .register((composer) => {

      composer.articles = composer.articles.map(article => {

        const { text, ...filteredArticle } = article;

        filteredArticle.comments = filteredArticle.comments.map(comment => {

          const { author, ...filteredComment } = comment;

          return filteredComment as typeof comment;

        });

        // Note setting to type of IArticle here so typescript
        // doesn't complain, this is because you are removing props
        // above so the shape changes so you may want to consider
        // setting the props you plan to strip as optional or make
        // everything a partial etc. I'll leave that to you to decide.
        return filteredArticle as typeof article;

      });

      return composer;

    })
    .run();

What's Next

From here to get where you want it's about string parsing. Keep in mind Lodash does support gets down into nested values in an array. You can see this here in the docs .

Given that you could leverage Lodash using both _.get _.omit... etc along with a little parsing using dot notation.

Done this very thing with permissions. As such I feel strongly you need to start with a simple api to process then from there make your map from either Glob like or SQL string to those helpers.