Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

bit of a newbie question(s) #44

Closed
ken-crozier opened this issue Mar 27, 2020 · 20 comments
Closed

bit of a newbie question(s) #44

ken-crozier opened this issue Mar 27, 2020 · 20 comments

Comments

@ken-crozier
Copy link

Hi, I'm relatively new to nfsen and have gotten to a point where I stuck. Sorry if this is very basic ... Through lots of reading I have nfcapd running creating files under /var/nfdump/profiles-data/live/bachman - i started it with

nfcapd -w -D -p 2025 -u netflow -g www-data -B 200000 -s 1 -z -I bachman -l /var/nfdump/profiles-data/live/bachman

Now onto where I'm stuck ... nfcapd has been running for about 10 hours so I would expect to see lots of data ... but this is what I get when I go to the browser ....

Screen Shot 2020-03-27 at 11 39 24 AM

And when I try ./cli.php status I get no response, it also says that a log should be created in /var/www/html/nfsen/backend ... but it's not - so what am i missing ??

Also in your readme, while it describes how to install nfsed-ng maybe a couple of lines on adding the netflow user and how to start nfcapd would be useful

thanks in advance
Ken
@Dona21
Copy link
Collaborator

Dona21 commented Mar 30, 2020

Hi Ken,

I will try to help you as much as possible, though I'm not an expert neither.

Understood that you are new to nfsen. Are you familiar with NetFlow ? If not, maybe start reading the wikipedia page https://en.wikipedia.org/wiki/NetFlow

Based on what you can read on the wikipedia page, you could imagine "nfsen-ng" as the web "Analysis application". It is actually web-based visualizer that uses nfdump which is the real analysis application.

nfsen-ng has nothing to do with the exporter part nor the collector part. The exporter could be a network device (software or hardware) and the collector in our case is "nfcapd" from "nfdump" suite.

To sum up, "nfsen-ng" will work only if you have collected data with your collector (nfcapd) which will collect data only if you have configured an exporter correctly.

My first question is : did you configure an exporter ? If you let the collector run without sending data to it, you will have no data to use.

Please post a tree output of your "/var/nfdump/profiles-data/live/bachman" with : tree -h /var/nfdump/profiles-data/live/bachman

What is the output of "cli.php" if you run it without arguments ? /your/path/to/www/nfsen-ng/backend/cli.php

Do you get output if you run nfdump with YOUR_FILE being a nfcapd file in that directory sudo nfdump -I -M bachman -r /var/nfdump/profiles-data/live/bachman/2020/03/27/YOUR_FILE

Thanks for your advice concerning the netflow user. Actually this is not covered here as this is part of the data collection which is accomplished with nfcapd from the nfdump suite and should probably be explained there. In any case, it is a good idea and if we manage to find a time slot with @mbolli we will implement some features, correct some bugs and work on the wiki to cover that part too. Currently we are overloaded with our respective jobs but the motivation to keep working on this project does exist ;-)

Looking forward to get your feedback and further help you

@ken-crozier
Copy link
Author

ken-crozier commented Mar 30, 2020 via email •
edited by Dona21
Loading

@Dona21
Copy link
Collaborator

Dona21 commented Mar 31, 2020

Hi Ken,

Based on your feedback, it seems that many things are not working and should be fixed.
For further use, may I ask you which OS you are using and which webserver ?

Below some observed points which might further help you :
(-) the nfdump output shows that there is no flow data in the file you selected
(-) the tree output shows that all nfcapd files are empty (no flow data)
(-) the tree output shows a structure which is currently not compatible with nfsen-ng -> see the first point under configuration that sates -> Note: nfsen-ng expects the profiles-data folder structure to be PROFILES_DATA_PATH/PROFILE/SOURCE/YYYY/MM/DD/nfcapd.YYYYMMDDHHII
(-) cli.php should return something when invoked without arguments, but this is the last thing to have a look at right now as you have no data and incompabile folder structure

Without having any idea about your infrastructure, I would do the following in this situation :
(-) As per your first message, you expect flow data to be captured by the collector on port 2025. Is the exporter sending the data on that port ?
(-) You could check the previous point in the exporter configuration first, but also check with a network packet capture like "wireshark" and/or check if a firewall is blocking this connection
(-) If flow data is sent by the collector and received by the network interface(s) of your collector, it might be a collector problem (structure, permissions, something else...)

Let's start with that

@ken-crozier
Copy link
Author

ken-crozier commented Mar 31, 2020 via email •
edited by Dona21
Loading

@Dona21
Copy link
Collaborator

Dona21 commented Mar 31, 2020

Hi Ken,

The comment you referred to is correct and does not need any correction. You are using the live profile already (seen in your path -> /var/nfdump/profiles-data/live/bachman).

If the files were not empty, your tree output would look like this (see my truncated output below). Instead of that, your tree output shows [276] in front of each file which corresponds to empty nfcapd files.

root@nfsen-ng: tree -h /var/nfsen/profiles-data/live/source1/2020/03
.
├── [ 11K] 01
│   ├── [ 18K] nfcapd.202003010000
│   ├── [ 17K] nfcapd.202003010005
│   ├── [ 15K] nfcapd.202003010010
│   ├── [ 18K] nfcapd.202003010015
│   ├── [ 15K] nfcapd.202003010020
│   ├── [ 16K] nfcapd.202003010025
│   ├── [ 18K] nfcapd.202003010030
│   ├── [ 18K] nfcapd.202003010035
│   ├── [ 15K] nfcapd.202003010040
│   ├── [ 16K] nfcapd.202003010045
│   ├── [ 14K] nfcapd.202003010050
│   ├── [ 16K] nfcapd.202003010055

Additionally, running the nfdump command would give you data if your files were not empty (see below example from my machine), which is much different from what you posted above.

root@nfsen-ng: nfdump -I -M source1 -r /var/nfsen/profiles-data/live/source1/2020/03/31/nfcapd.202003311345
Can't stat 'source1': No such file or directory
Ident: source1
Flows: 1116
Flows_tcp: 12
Flows_udp: 446
Flows_icmp: 71
Flows_other: 587
Packets: 2319
Packets_tcp: 651
Packets_udp: 850
Packets_icmp: 90
Packets_other: 728
Bytes: 451861
Bytes_tcp: 128992
Bytes_udp: 206805
Bytes_icmp: 9546
Bytes_other: 106518
First: 1585655034
Last: 1585655395
msec_first: 444
msec_last: 130
Sequence failures: 132

Therefore, you have have issues with your collector. Check the nfdump (and related nfcapd) github page (https://github.com/phaag/nfdump) to have a working instance, as nfsen-ng will work only once these tools are setup correctly

@ken-crozier
Copy link
Author

ken-crozier commented Apr 1, 2020 via email

@Dona21
Copy link
Collaborator

Dona21 commented Apr 3, 2020

Hi Ken,

Glad to hear that you managed to have the collector working.

The profiles-data folder structure however is still wrong (by wrong I mean not compatible with nfsen-ng).

nfsen-ng expects the profiles-data folder structure to be PROFILES_DATA_PATH/PROFILE/SOURCE/YYYY/MM/DD/nfcapd.YYYYMMDDHHII

The captured data in your case is PROFILES_DATA_PATH/PROFILE/SOURCE/YYYY/MM/DD/HH/nfcapd.YYYYMMDDHHII

Based on your first comment this should be easily fixed by changing your lower case "-s" parameter to upper case "-S"

this : nfcapd -w -D -p 2025 -u netflow -g www-data -B 200000 -s 1 -z -I bachman -l /var/nfdump/profiles-data/live/bachman

becomes that : nfcapd -w -D -p 2025 -u netflow -g www-data -B 200000 -S 1 -z -I bachman -l /var/nfdump/profiles-data/live/bachman

Concerning the message you are getting when executing cli.php, could you post the content of this file /etc/php/7.2/mods-available/rrd.ini

@ken-crozier
Copy link
Author

ken-crozier commented Apr 3, 2020 via email

@mbolli
Copy link
Owner

mbolli commented Apr 3, 2020

Hi Ken,

the contents of rrd.ini should only be extension=rrd.so.
and phpenmod rrd is a command to execute in the terminal.
Don't forget to service apache2 restart after the command above.

enjoy your weekend!
Michael

@ken-crozier
Copy link
Author

ken-crozier commented Apr 3, 2020 via email

@mbolli
Copy link
Owner

mbolli commented Apr 3, 2020

Hmm that's weird.
Can you confirm that there's a file 20-rrd.ini (or similar) in both of these folders:

  • /etc/php/7.2/cli/conf.d
  • /etc/php/7.2/apache2/conf.d

@ken-crozier
Copy link
Author

ken-crozier commented Apr 3, 2020 via email

@mbolli
Copy link
Owner

mbolli commented Apr 3, 2020

What about php -i | grep rrd ?

@ken-crozier
Copy link
Author

ken-crozier commented Apr 3, 2020 via email

@mbolli
Copy link
Owner

mbolli commented Apr 3, 2020 •
edited
Loading

Ok so the php rrd module is definitely not loaded (at least for cli profile)
My output:

mbolli@newdev:/var/www$ php -i | grep rrd
/etc/php/7.2/cli/conf.d/20-rrd.ini,
rrd
rrd tool module => enabled
rrd tool module version => 2.0.1
rrdtool library version => 1.7.0

There lies the culprit, but I don't know how to help from here... You got to somehow enable the rrd module (for cli and apache2 profiles).

@Dona21
Copy link
Collaborator

Dona21 commented Apr 3, 2020

Hi Ken,

Your *rrd.ini files are wrong.
If you cat the files the output should be extension=rrd.so instead of add extension=rrd.so.
Modify the files accordingly and then restart apache webesrver with systemctl restart apache2

This particular issue should be fixed then.

@ken-crozier
Copy link
Author

ken-crozier commented Apr 3, 2020 via email

@Dona21
Copy link
Collaborator

Dona21 commented Apr 3, 2020

I would personally delete the content directory content and start everything in a clean manner now that you know how to have a running instance. However, if for any reason you prefer to keep the data, it might be better to move the data rather than make a copy. nfsen-ng should work with that structure then. Also, do not mix nfsen and nfsen-ng. The first is the application written by the same person who created the whole nfdump suite and the second is the one we are talking about.

In order to avoid other people having the same issue, I will slightly change the instructions to make it less error prone.

I think your issue is solved, please consider closing it. Have a good weekend

@Dona21 Dona21 closed this as completed Apr 22, 2020
@Dona21 Dona21 mentioned this issue Dec 23, 2020
Closed
@Dona21 Dona21 mentioned this issue Oct 6, 2022
Closed
@pra838
Copy link

pra838 commented Aug 6, 2023

image No graphs.

root@nfdump:/var/nfdump/profiles-data/live/ExportFlow# tree -h
[4.0K] .
├── [4.0K] 2023
│   └── [4.0K] 08
│   ├── [4.0K] 01
│   │   ├── [ 276] nfcapd.202308011525
│   │   ├── [ 276] nfcapd.202308011530
│   │   ├── [ 276] nfcapd.202308011535
│   │   ├── [ 276] nfcapd.202308011540
│   │   ├── [ 276] nfcapd.202308011545
│   │   ├── [ 276] nfcapd.202308011550
│   │   ├── [ 276] nfcapd.202308011555
│   │   ├── [ 276] nfcapd.202308011600
│   │   ├── [ 276] nfcapd.202308011605
│   │   ├── [ 276] nfcapd.202308011610
│   │   ├── [ 276] nfcapd.202308011615
│   │   ├── [ 276] nfcapd.202308011620
│   │   ├── [ 276] nfcapd.202308011625
│   │   ├── [ 276] nfcapd.202308011630
│   │   ├── [ 276] nfcapd.202308011635
│   │   ├── [ 276] nfcapd.202308011640

@Dona21
Copy link
Collaborator

Dona21 commented Aug 6, 2023

Hi @pra838

Referring to this answer you will notice that your tree output shows [276] in front of each file which corresponds to empty nfcapd files. Therefore it is normal that no graphs are available because you have no data collected. You should have your nfcapd collector up, running and working/configured correctly before using nfsen-ng.

Cheers

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@mbolli @Dona21 @pra838 @ken-crozier