nifi stream csv 文件到 hdfs

Question

设想

我正在尝试使用 NiFi 对商店实时更新 csv 文件到 HDFS 进行研究。 我有 2 个文件， kafka2hdfs.csv和test_csv.csv 。 两者都包含相同的信息

a,b,c,d,e
a1,b1,c1,d1,e1
a2,b2,c2,d2,e2
a3,b3,c3,d3,e3
a4,b4,c4,d4,e4
a5,b5,c5,d5,e5
a6,b6,c6,d6,e6
a7,b7,c7,d7,e7
a8,b8,c8,d8,e8

我正在使用filebeat将文件传输到nifi 。 只有几行filebeat很快就会真正触底。 文件节拍

2020-07-15T16:07:32.061+0700    DEBUG   [harvester]     log/log.go:107  End of file reached: C:\Program Files\Filebeat\test-kafka\test_csv.csv; Backoff now.
2020-07-15T16:07:32.090+0700    DEBUG   [registrar]     registrar/registrar.go:404      Registry file updated. 2 states written.
2020-07-15T16:07:32.091+0700    DEBUG   [registrar]     registrar/registrar.go:356      Processing 1 events
2020-07-15T16:07:32.093+0700    DEBUG   [registrar]     registrar/registrar.go:326      Registrar state updates processed. Count: 1
2020-07-15T16:07:32.093+0700    DEBUG   [registrar]     registrar/registrar.go:411      Write registry file: C:\Program Files\Filebeat\data\registry\filebeat\data.json (2)
2020-07-15T16:07:32.140+0700    DEBUG   [registrar]     registrar/registrar.go:404      Registry file updated. 2 states written.
2020-07-15T16:07:33.013+0700    INFO    [publisher_pipeline_output]     pipeline/output.go:101  Connecting to backoff(async(tcp://10.64.2.236:5044))
2020-07-15T16:07:33.013+0700    DEBUG   [logstash]      logstash/async.go:119   connect
2020-07-15T16:07:33.031+0700    INFO    [publisher_pipeline_output]     pipeline/output.go:111  Connection to backoff(async(tcp://10.64.2.236:5044)) established
2020-07-15T16:07:33.036+0700    DEBUG   [logstash]      logstash/async.go:171   17 events out of 17 events sent to logstash host 10.64.2.236:5044. Continue sending
2020-07-15T16:07:33.042+0700    DEBUG   [harvester]     log/log.go:107  End of file reached: C:\Program Files\Filebeat\test-kafka\kafka2hdfs.csv; Backoff now.

并成功在 hdfs 上创建文件

hadoop@ambari:~$ hdfs dfs -ls /user/test
Found 2 items
-rw-r--r--   3 nifi hdfs        114 2020-07-15 16:07 /user/test/kafka2hdfs.csv
-rw-r--r--   3 nifi hdfs        129 2020-07-15 16:06 /user/test/test_csv.csv
hadoop@ambari:~$ hdfs dfs -cat /user/test/test_csv.csv
a,b,c,d,e
a1,b1,c1,d1,e1
a2,b2,c2,d2,e2
a3,b3,c3,d3,e3
a4,b4,c4,d4,e4
a5,b5,c5,d5,e5
a6,b6,c6,d6,e6
a7,b7,c7,d7,e7
a8,b8,c8,d8,e8

现在我正在尝试将test_csv.csv文件更新为

a,b,c,d,e
a1,b1,c1,d1,e1
a2,b2,c2,d2,e2
a3,b3,c3,d3,e3
a4,b4,c4,d4,e4
a5,b5,c5,d5,e5
a6,b6,c6,d6,e6
a7,b7,c7,d7,e7
a8,b8,c8,d8,e8
9,9,9,9,9
10,10,10,10,10

然后重新运行它，但在 hdfs 上它没有更新。 Filebeat实际发送的数据

2020-07-15T16:26:54.892+0700    DEBUG   [processors]    processing/processors.go:187      Publish event: {
  "@timestamp": "2020-07-15T09:26:54.892Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "7.7.1"
  },
  "message": "10,10,10,10,10",
  "log": {
    "offset": 150,
    "file": {
      "path": "C:\\Program Files\\Filebeat\\test-kafka\\test_csv.csv"
    }
  },
  "input": {
    "type": "log"
  },
  "ecs": {
    "version": "1.5.0"
  },
  "host": {
    "name": "name"
  },
  "agent": {
    "hostname": "name",
    "id": "5e2fec03-bbdc-4f91-acc9-4ab36c7268db",
    "version": "7.7.1",
    "type": "filebeat",
    "ephemeral_id": "e1f3e6e0-f4f5-4f68-ad1d-8bb1767e4d4a"
  }
}

hadoop@ambari:~$ hdfs dfs -cat /user/test/test_csv.csv
a,b,c,d,e
a1,b1,c1,d1,e1
a2,b2,c2,d2,e2
a3,b3,c3,d3,e3
a4,b4,c4,d4,e4
a5,b5,c5,d5,e5
a6,b6,c6,d6,e6
a7,b7,c7,d7,e7
a8,b8,c8,d8,e8

建筑学

CSV - Filebeat - NiFi - HDFS

尼菲

期待

我以为 nifi 会将 hdfs 中较早生成的文件更新为最新编辑。 但事实并非如此。

9,9,9,9,9

但在 hdfs 中的 test_csv.csv 上没有hdfs

问题

1. NiFi可以这样做吗？
2. 我错过了任何处理器吗？