Coletor de fluxo unificado ElastiFlow
O ElastiFlow Unified Flow Collector recebe, decodifica, transforma, normaliza, traduz e enriquece registros de fluxo de rede e telemetria enviados de dispositivos e aplicativos de rede usando IPFIX, Netflow e sFlow. Os registros resultantes podem ser enviados para diversas plataformas e serviços.

Requisitos de sistema acesse aqui.

Com seu Debian 11 (Bullseye) 64 bits. Instalação mínima pronta

Existem outras formas de configurar, mas aqui tentei passar a forma mais simples possível para que qualquer um consiga insta-lo.

# su - 

Instale os pacotes básicos necessários

# apt install wget apt-transport-https gnupg unzip curl net-tools

Recomendo a instalacao dos firmwares binário para vários drivers do kernel Linux, para isso adicione contrib non-free ao seu repositório.

# vim /etc/apt/sources.list

deb http://deb.debian.org/debian/ bullseye main contrib non-free
deb-src http://deb.debian.org/debian/ bullseye main contrib non-free

deb http://security.debian.org/debian-security bullseye-security main contrib non-free
deb-src http://security.debian.org/debian-security bullseye-security main contrib non-free

# bullseye-updates, to get updates before a point release is made;
# see https://www.debian.org/doc/manuals/debian-reference/ch02.en.html#_updates_and_backports
deb http://deb.debian.org/debian/ bullseye-updates main contrib non-free
deb-src http://deb.debian.org/debian/ bullseye-updates main contrib non-free

# apt update
# apt install firmware-linux firmware-linux-free firmware-linux-nonfree

Tuning no kernel

# echo "vm.swappiness=10" >> /etc/sysctl.conf
# echo "vm.max_map_count=262144" > /etc/sysctl.d/70-elasticsearch.conf
# cat <<EOF >/etc/sysctl.d/60-net.conf
net.core.netdev_max_backlog=4096
net.core.rmem_default=262144
net.core.rmem_max=67108864
net.ipv4.udp_rmem_min=131072
net.ipv4.udp_mem=2097152 4194304 8388608
EOF

# sysctl -w vm.max_map_count=262144 && \
sysctl -w net.core.netdev_max_backlog=4096 && \
sysctl -w net.core.rmem_default=262144 && \
sysctl -w net.core.rmem_max=67108864 && \
sysctl -w net.ipv4.udp_rmem_min=131072 && \
sysctl -w net.ipv4.udp_mem='2097152 4194304 8388608'

Reinicie para que os novos módulos sejam carregado.

# reboot

:: Elastic Stack ::

Será necessário ter o Java instalado.

# su -
# apt install default-jre

Adicione o repositório elastic.

# wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | apt-key add -
# echo "deb https://artifacts.elastic.co/packages/8.x/apt stable main" |  tee -a /etc/apt/sources.list.d/elastic.list
# apt update

:: Elasticsearch ::

Elasticsearch é um mecanismo de pesquisa baseado em texto completo.

# apt install elasticsearch

Se uma JVM for iniciada com tamanhos de heap inicial e máximo desiguais, ela poderá pausar conforme o heap da JVM for redimensionado durante o uso do sistema. Por esse motivo, é melhor iniciar a JVM com os tamanhos de heap inicial e máximo configurados com valores iguais.

Adicione o arquivo e defina e heap.optionsa cerca de um terço da memória do sistema, mas não exceda . Para este exemplo, usaremos 12 GB dos 32 GB de memória disponíveis para heap da JVM.

# echo -e "-Xms12g\n-Xmx12g" > /etc/elasticsearch/jvm.options.d/heap.options

Os limites do sistema aumentados devem ser especificados em um systemd.

# mkdir /etc/systemd/system/elasticsearch.service.d
# cat <<EOF >/etc/systemd/system/elasticsearch.service.d/elasticsearch.conf
[Service]
LimitNOFILE=131072
LimitNPROC=8192
LimitMEMLOCK=infinity
LimitFSIZE=infinity
LimitAS=infinity
EOF

Após instalação vamos alterar network.host para ouvir apenas localhost, em seguida ativar o serviço e inicia-lo.

# sed -i 's/#cluster.name: my-application/cluster.name: elastiflow/' /etc/elasticsearch/elasticsearch.yml
# sed -i 's/#network.host: 192.168.0.1/network.host: 127.0.0.1/' /etc/elasticsearch/elasticsearch.yml
# sed -i 's/xpack.security.enabled: true/xpack.security.enabled: false/' /etc/elasticsearch/elasticsearch.yml
# echo "indices.query.bool.max_clause_count: 8192" >> /etc/elasticsearch/elasticsearch.yml
# echo "search.max_buckets: 250000" >> /etc/elasticsearch/elasticsearch.yml

# systemctl daemon-reload
# systemctl enable elasticsearch
# systemctl start elasticsearch
# systemctl status elasticsearch

● elasticsearch.service - Elasticsearch
     Loaded: loaded (/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/elasticsearch.service.d
             └─elasticsearch.conf
     Active: active (running) since Fri 2022-02-04 11:34:39 -03; 53s ago
       Docs: https://www.elastic.co
   Main PID: 4701 (java)
      Tasks: 60 (limit: 38490)
     Memory: 12.7G
        CPU: 52.548s
     CGroup: /system.slice/elasticsearch.service
             ├─4701 /usr/share/elasticsearch/jdk/bin/java -Xshare:auto -Des.networkaddress.cache.ttl=60 .....
             └─4893 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller

fev 04 11:34:23 debian systemd[1]: Starting Elasticsearch...
fev 04 11:34:39 debian systemd[1]: Started Elasticsearch.

Cheque se o mesmo esta funcionando

# curl -X GET "http://127.0.0.1:9200"

Retorno algo como….

{
  "name" : "debianFlows",
  "cluster_name" : "elastiflow",
  "cluster_uuid" : "mgKF5EP-SnKbbULl3D6aXQ",
  "version" : {
    "number" : "8.4.2",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "89f8c6d8429db93b816403ee75e5c270b43a940a",
    "build_date" : "2022-09-14T16:26:04.382547801Z",
    "build_snapshot" : false,
    "lucene_version" : "9.3.0",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}

:: kibana ::

Kibana é uma plataforma de análise e visualização usada para exibir os painéis do ElastiFlow.

# apt install kibana

Após instalação vamos alterar server.host, em seguida ativar o serviço e inicia-lo.

# echo "telemetry.enabled: false" >> /etc/kibana/kibana.yml
# echo "telemetry.optIn: false" >> /etc/kibana/kibana.yml
# echo "newsfeed.enabled: false" >> /etc/kibana/kibana.yml
# echo "server.host: \"0.0.0.0\"" >> /etc/kibana/kibana.yml
# echo "server.maxPayload: 8388608" >> /etc/kibana/kibana.yml
# echo "elasticsearch.requestTimeout: 132000" >> /etc/kibana/kibana.yml
# echo "elasticsearch.shardTimeout: 120000" >> /etc/kibana/kibana.yml

# vim /etc/kibana/kibana.yml

Ajuste para o endereço da sua URL(Domínio ou IP) pública, exemplo:

server.publicBaseUrl: "http://192.168.0.30:5601"

# systemctl daemon-reload
# systemctl enable kibana
# systemctl start kibana
# systemctl status kibana

● kibana.service - Kibana
     Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2022-02-03 17:45:24 -03; 3s ago
       Docs: https://www.elastic.co
   Main PID: 4638 (node)
      Tasks: 11 (limit: 6548)
     Memory: 138.6M
        CPU: 4.339s
     CGroup: /system.slice/kibana.service
             └─4638 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli/dist\ 
                                           --logging.dest=/var/log/kibana/kibana.log\ 
                                           --pid.file=/run/kibana/kibana.pi>

fev 03 17:45:24 deb11 systemd[1]: Started Kibana.

:: ElastiFlow ::

ElastiFlow fornece coleta e visualização de dados de fluxo de rede usando o Elastic Stack.

# apt install libpcap-dev
# wget https://elastiflow-packages.s3.amazonaws.com/flow-collector/flow-collector_5.6.0_linux_amd64.deb
# dpkg -i flow-collector_5.6.0_linux_amd64.deb
# apt install -f
# systemctl daemon-reload 
# systemctl enable flowcoll
# systemctl start flowcoll
# systemctl status flowcoll

● flowcoll.service - ElastiFlow Unified Flow Collector
     Loaded: loaded (/etc/systemd/system/flowcoll.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/flowcoll.service.d
             └─flowcoll.conf
     Active: active (running) since Fri 2022-02-04 13:49:34 -03; 8s ago
       Docs: https://docs.elastiflow.com
   Main PID: 10763 (flowcoll)
      Tasks: 9 (limit: 38490)
     Memory: 33.0M
        CPU: 105ms
     CGroup: /system.slice/flowcoll.service
             └─10763 /usr/share/elastiflow/bin/flowcoll

# wget https://raw.githubusercontent.com/elastiflow/elastiflow_for_elasticsearch/master/kibana/kibana-7.14.x-codex-dark.ndjson
# curl -XPOST "http://127.0.0.1:5601/api/saved_objects/_import?overwrite=true"\
 -k -H "kbn-xsrf: true" -H "securitytenant: global" --form file=@kibana-7.14.x-codex-dark.ndjson

No retornos dos comando curl certifique-se de ter no inicio a mensagem “success”:true”

{"successCount":229,"success":true,"warnings":[],"successResults...............

Configurações Avançadas Recomendadas do Kibana

Acesse seu Kibana no navegador. http://____:5601

Realizando esses ajustes avançados o Kibana produzirá uma experiência mais amigável ao usar o ElastiFlow. Essas configurações são feitas no Kibana, em Stack Management -> Advanced Settings.
Use a busca para localizar e ajustar os seguintes valores:

Pin filters by default: On
Highlight results : Off
Store URLs in session storage: On
Dark mode: On
Day of week: Monday
Formatting locale: Portuguese (Brazil)
Number format: 0,0.[00]
Percent format: 0,0.[00]%
Time filter defaults:

{
  "from": "now-1h/m",
  "to": "now"
}

Time filter quick ranges:

[
  {
    "from": "now-15m/m",
    "to": "now/m",
    "display": "Last 15 minutes"
  },
  {
    "from": "now-30m/m",
    "to": "now/m",
    "display": "Last 30 minutes"
  },
  {
    "from": "now-1h/m",
    "to": "now/m",
    "display": "Last 1 hour"
  },
  {
    "from": "now-2h/m",
    "to": "now/m",
    "display": "Last 2 hours"
  },
  {
    "from": "now-4h/m",
    "to": "now/m",
    "display": "Last 4 hours"
  },
  {
    "from": "now-12h/m",
    "to": "now/m",
    "display": "Last 12 hours"
  },
  {
    "from": "now-24h/m",
    "to": "now/m",
    "display": "Last 24 hours"
  },
  {
    "from": "now-48h/m",
    "to": "now/m",
    "display": "Last 48 hours"
  },
  {
    "from": "now-7d/m",
    "to": "now/m",
    "display": "Last 7 days"
  },
  {
    "from": "now-30d/m",
    "to": "now/m",
    "display": "Last 30 days"
  },
  {
    "from": "now-60d/m",
    "to": "now/m",
    "display": "Last 60 days"
  },
  {
    "from": "now-90d/m",
    "to": "now/m",
    "display": "Last 90 days"
  }
]

Ajustes finos:

# vim /etc/systemd/system/flowcoll.service.d/flowcoll.conf

Ajustes ELASTICSEARCH

Environment="EF_FLOW_OUTPUT_ELASTICSEARCH_ENABLE=true"
Environment="EF_FLOW_OUTPUT_ELASTICSEARCH_TIMESTAMP_SOURCE=end"
Environment="EF_FLOW_OUTPUT_ELASTICSEARCH_INDEX_TEMPLATE_ILM_LIFECYCLE=elastiflow"
#Environment="EF_FLOW_OUTPUT_ELASTICSEARCH_USERNAME=elastic"
#Environment="EF_FLOW_OUTPUT_ELASTICSEARCH_PASSWORD=changeme"

Ajustes DNS

Environment="EF_FLOW_DECODER_ENRICH_DNS_ENABLE=true"
Environment="EF_FLOW_DECODER_ENRICH_DNS_NAMESERVER_IP=1.1.1.1"

Ajustes MAXMIND

Environment="EF_FLOW_DECODER_ENRICH_MAXMIND_ASN_ENABLE=true" 
Environment="EF_FLOW_DECODER_ENRICH_MAXMIND_GEOIP_ENABLE=true"
Environment="EF_FLOW_DECODER_ENRICH_MAXMIND_GEOIP_LANG=pt-BR"
Environment="EF_FLOW_DECODER_ENRICH_MAXMIND_ASN_PATH=maxmind/GeoLite2-ASN.mmdb"
Environment="EF_FLOW_DECODER_ENRICH_MAXMIND_GEOIP_PATH=maxmind/GeoLite2-City.mmdb"

Base MAXMIND.

# cd /etc/elastiflow/maxmind
# wget https://git.io/GeoLite2-ASN.mmdb
# wget https://git.io/GeoLite2-City.mmdb
# wget https://git.io/GeoLite2-Country.mmdb

Reiniciando os serviços

# systemctl daemon-reload
# systemctl restart flowcoll
# systemctl status flowcoll
# systemctl restart elasticsearch kibana
# systemctl status elasticsearch kibana

Configure seu router para enviar os flows

Bonus: Huawei NE Netstream (NetFlow)

ip netstream as-mode 32
ip netstream timeout active 1
ip netstream timeout inactive 15
ip netstream export version 9 origin-as
ip netstream export index-switch 32
ip netstream export template timeout-rate 2
ip netstream sampler fix-packets 128 inbound
ip netstream sampler fix-packets 128 outbound
ip netstream export source IPv4_LOOPBACK_ORIGEM
ip netstream export host IPv4_SERVIDOR 9995
ip netstream export template option sampler
ip netstream export template option application-label
#
ipv6 netstream as-mode 32
ipv6 netstream timeout active 1
ipv6 netstream timeout inactive 15
ipv6 netstream export version 9 origin-as
ipv6 netstream export index-switch 32
ipv6 netstream export template timeout-rate 2
ipv6 netstream sampler fix-packets 128 inbound
ipv6 netstream sampler fix-packets 128 outbound
ipv6 netstream export source IPv4_LOOPBACK_ORIGEM
ipv6 netstream export host IPv4_SERVIDOR 9995
ipv6 netstream export template option sampler
 
# CONFIG NO SLOT -- slot 9 ou 10 NE8000 / slot 3 NE40
# EXECUTE O CMD NO ROOT, COMANDO NÃO FUNCIONA NO VIRTUAL SYSTEM
slot 9
 ip netstream sampler to slot self
 ipv6 netstream sampler to slot self
 
# APLICAR APENAS NAS INTERFACES UPSTREAM (LINK IP, PTT, IX...)
ip netstream inbound
ip netstream outbound
ipv6 netstream inbound
ipv6 netstream outbound
 
# EM CASO DE INTERFACES DE CDN INTERNO, APLICAR:
ip netstream inbound
ipv6 netstream inbound
 
# EM CASO DE INTERFACE BILATERAL (ONDE TANTO RECEBE QUANTO ENVIA CONTEÚDO), APLICAR:
ip netstream inbound
ipv6 netstream inbound

Bonus: Mikrotik

/ip traffic-flow
set active-flow-timeout=1m cache-entries=4k enabled=yes interfaces=INTERFACES_COLETAR
/ip traffic-flow target 
add dst-address=IP_SERVER_ASSTATS port=9995 src-address=IP_ORIGEM version=9

Verificar se os pacotes estão chegando na porta 9995

# apt install tcpdump
# tcpdump -i ens18 -n udp port 9995 -T cnfp

Acesse eu Kibana e seja feliz!

Escolha sua Dash

Gostou e quer me ajudar a manter o blog?

Se quiser fazer uma doação para o café ficarei muito feliz pelo seu reconhecimento!

Se não puder doar pode deixar seu agradecimento nos comentário também ficarei feliz em saber que ajudei. Se tiver qualquer pergunta deixe-a também. Se preferir entrar em Contato clique aqui.

Abraço!

Fonte:
https://docs.elastiflow.com

O post Aprenda instalar o ElastiFlow, uma poderosa ferramenta para análise de tráfego de rede no Debian 11 Bullseye apareceu primeiro em Remontti.