Application Security Community

 View Only
Back to discussions
Expand all | Collapse all

Error : Incapsula Log downloader Script

  • 1.  Error : Incapsula Log downloader Script

    Posted 11-05-2020 07:29

    Hi,

    I'm running the downloader script on Windows Server 2012R2 with Python 3.8.6 version.
    Some log file cause below error.
    Has anyone experience this before and how to fix it?

    Thank you.
    Worachat Sarsa

    2020-11-04 15:00:07,760 INFO Successfully downloaded file from URL https://logs1.incapsula.com/6827_1XXX7/6827_21977.log
    2020-11-04 15:00:07,760 WARNING Keys Dir: C:\incapsula\logs-downloader-master2-Test\config\keys
    2020-11-04 15:00:07,776 INFO Syslog enabled, using UDP
    2020-11-04 15:00:14,870 INFO Saving file 6827_21977.log locally to the 'fail' folder 'charmap' codec can't encode character '\u014c' in position 0: character maps to Traceback (most recent call last):
    File "C:\incapsula\logs-downloader-master2-Test\script\LogsDownloader.py", line 222, in handle_file
    self.handle_log_decrypted_content(logfile, decrypted_file)
    File "C:\incapsula\logs-downloader-master2-Test\script\LogsDownloader.py", line 276, in handle_log_decrypted_content
    local_file.writelines(decrypted_file)
    File "C:\Python38\lib\encodings\cp874.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
    UnicodeEncodeError: 'charmap' codec can't encode character '\u014c' in position 0: character maps to <undefined>


    #CloudWAF(formerlyIncapsula)

    ------------------------------
    Worachat Sarsa
    Exclusive Networks TH
    ------------------------------


  • 2.  RE: Error : Incapsula Log downloader Script

    Posted 11-05-2020 07:37
    @Worachat Sarsa, 

    Have you checked out Imperva Github? Specifically checkout incapsula-logs-downloader. Let  me know if this helps. 


    ​

    ------------------------------
    Christopher Detzel
    Community Manager
    Imperva
    ------------------------------



  • 3.  RE: Error : Incapsula Log downloader Script

    Posted 11-06-2020 00:30
    Hi @Christopher Detzel​,

    Thank you for your reply.
    The script I used is actually from Githup that you were mentioned. I also post the issue to it as well : https://github.com/imperva/incapsula-logs-downloader/issues/19

    However, As I search It seems to be window's terminal issue that not support some encoded character

    So I try to change the script, line 255 as below, to avoid this issue.
    from : decrypted_file = decrypted_file.decode('utf-8')
    to : decrypted_file = decrypted_file.decode('utf-8', 'ignore')

    But it still running the same error.
    Please help suggest.

    Thank you so much in advanced.

    the log which i test today is below :
    2020-11-06 11:01:49,027 INFO Sleeping for 2 seconds before fetching the next logs file
    2020-11-06 11:01:57,216 INFO Successfully downloaded file from URL https://logs1.incapsula.com/6827_XXXXX/6827_26545.log
    2020-11-06 11:01:57,216 WARNING Keys Dir: C:\incapsula\logs-downloader-master2-Test\config\keys
    2020-11-06 11:01:57,232 INFO Syslog enabled, using UDP
    2020-11-06 11:01:58,966 INFO Saving file 6827_26545.log locally to the 'fail' folder 'charmap' codec can't encode character '\xe3' in position 0: character maps to Traceback (most recent call last):
    File "C:\incapsula\logs-downloader-master2-Test\script\LogsDownloader.py", line 222, in handle_file
    self.handle_log_decrypted_content(logfile, decrypted_file)
    File "C:\incapsula\logs-downloader-master2-Test\script\LogsDownloader.py", line 277, in handle_log_decrypted_content
    local_file.writelines(decrypted_file)
    File "C:\Python38\lib\encodings\cp874.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
    UnicodeEncodeError: 'charmap' codec can't encode character '\xe3' in position 0: character maps to

    2020-11-06 11:01:58,966 ERROR Failed to download file 6827_26545.log. Error is - write() argument must be str, not bytes , Traceback (most recent call last):
    File "C:\incapsula\logs-downloader-master2-Test\script\LogsDownloader.py", line 222, in handle_file
    self.handle_log_decrypted_content(logfile, decrypted_file)
    File "C:\incapsula\logs-downloader-master2-Test\script\LogsDownloader.py", line 277, in handle_log_decrypted_content
    local_file.writelines(decrypted_file)
    File "C:\Python38\lib\encodings\cp874.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
    UnicodeEncodeError: 'charmap' codec can't encode character '\xe3' in position 0: character maps to

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
    File "C:\incapsula\logs-downloader-master2-Test\script\LogsDownloader.py", line 143, in get_log_files
    success = self.handle_file(next_file)
    File "C:\incapsula\logs-downloader-master2-Test\script\LogsDownloader.py", line 233, in handle_file
    file.write(result[1])
    TypeError: write() argument must be str, not bytes

    2020-11-06 11:02:46,998 INFO Successfully downloaded file from URL https://logs1.incapsula.com/6827_XXXXX/6827_26545.log



    ------------------------------
    Worachat Sarsa
    Exclusive Networks TH
    ------------------------------

    Original Message


  • 4.  RE: Error : Incapsula Log downloader Script

    Posted 11-10-2020 11:42
    @Worachat Sarsa did you get this resolved and was it something with the Windows Server? ​

    ------------------------------
    Joe Moore
    Director of Technology
    imperva
    ------------------------------

    Original Message


  • 5.  RE: Error : Incapsula Log downloader Script

    Posted 11-10-2020 20:13
    Hi @Joseph Ryan Moore,

    Thank you very much for your reply.

    ​I am facing the same issue and still finding the solution.

    ------------------------------
    Worachat Sarsa
    Exclusive Networks TH
    ------------------------------

    Original Message


  • 6.  RE: Error : Incapsula Log downloader Script

    Posted 11-10-2020 20:49
    @Worachat Sarsa​ two questions:
    1. Are you compressing the logs in the UI config settings
    2. Are you encrypting the logs with your key in the UI settings?

    I know that the log downloader requires the below for encryption but we can work around this if they are not being installed correctly on the Windows box:
    • pycrypto
    • M2Crypto
    pip3 install -r requirements.txt

    ------------------------------
    Joe Moore
    Director of Technology
    imperva
    ------------------------------

    Original Message


  • 7.  RE: Error : Incapsula Log downloader Script

    Posted 11-10-2020 20:56
    @Joseph Ryan Moore,

    To answer your question :
    ​1. Are you compressing the logs in the UI config settings?
    - Yes, compression is enabled.
    2. Are you encrypting the logs with your key in the UI settings?
    - We tested on both encryption enabled and disabled the error is still the same. At present it is disabled.

    All 3 required dependencies are installed on the system.
    - pycrypto
    - M2Crypto
    - urllib3




    ------------------------------
    Worachat Sarsa
    Exclusive Networks TH
    ------------------------------

    Original Message


  • 8.  RE: Error : Incapsula Log downloader Script

    Posted 11-10-2020 22:27
    @Worachat Sarsa ​ can you please confirm what account you are working with and share with me your config file?

    ------------------------------
    Joe Moore
    Director of Technology
    imperva
    ------------------------------

    Original Message


  • 9.  RE: Error : Incapsula Log downloader Script
    Best Answer

    Posted 11-11-2020 10:24
    After working with @Worachat Sarsa on this, we came to find that this is related to the encoding when trying to write the file to disk.
    I believe this is because the computer is Asia and possibly something with the default encoding settings, HOWEVER!!!

    The fix that was applied is online 275 in the .py file where I added the encoding option on the open method. 
    local_file = open(self.config.PROCESS_DIR + filename, "a+", encoding="UTF-8")
    @Aaron Seibert​​ can you add this to the repo?

    ------------------------------
    Joe Moore
    Director of Technology
    imperva
    ------------------------------



Related Content